ARL-Next

<div align="center"> # ARL-Next **AI Native Automation Asset Detection and Vulnerability Monitoring Platform** <p> <a href="https://hub.docker.com/"><img src="https://img.shields.io/badge/docker-ready-blue.svg?style=flat-square&logo=docker" alt="Docker"></a> <img src="https://img.shields.io/badge/python-3.8%2B-blue?style=flat-square&logo=python" alt="Python"> <img src="https://img.shields.io/badge/node.js-18%2B-green?style=flat-square&logo=node.js" alt="Node.js"> <img src="https://img.shields.io/badge/vue-3.x-4fc08d?style=flat-square&logo=vuedotjs" alt="Vue"> <img src="https://img.shields.io/badge/MCP-Ready-purple?style=flat-square" alt="MCP"> </p> </div> <br/> --- ## 💡 What is ARL-Next? **ARL-Next** is a modern reconstruction of ARL (Asset Reconnaissance Beacon), now fully evolved into an **AI Native Automation Asset Detection and Vulnerability Monitoring Platform**. Key highlights include: * **🤖 AI Native Architecture (New!)**: Native integration with MCP, enabling AI Agent to directly take over scheduling and retrieval, and start "conversational security operations". (👉 [**Explore MCP Usage and Configuration Guide**](./mcp-server/README.md)) * **🚀 Engine and Architecture Microservices**: Eliminate PhantomJS, upgrade Chromium and Nuclei engines; decouple OSINT and headless browser microservices, support memory caching for fingerprint recognition, and greatly improve scanning performance. * **🌐 Multi-dimensional Asset Loop**: Deeply integrate ICP and Tianyancha, fully automate the discovery of enterprise websites, apps, mini-programs, and other multi-dimensional assets. * **🛡️ Continuous Threat Monitoring**: Built-in GitHub latest CVE vulnerability tracking and code leakage real-time monitoring, realizing threat intelligence ahead. * **⚡ Extreme Deployment and Operation**: Support one-click deployment, container self-healing, smooth hot update, and Basic Auth forced defense; development environment supports rapid hot reload. --- ## 📸 Interface Preview * **Global Dashboard**: Real-time display of system resource consumption, background task status, multi-dimensional risk statistics, and latest log stream. <br><img src="./img/dashboard.png" alt="Dashboard" width="800"><br> * **OSINT Asset Detection**: Support ICP and Tianyancha and other intelligence association retrieval, one-click synchronization of enterprise multi-dimensional assets and seamless task issuance. <br><img src="./img/enterprise-asset-search1.png" alt="Asset Detection" width="800"><br> * **Task and Fingerprint Management**: Support task full life cycle tracking, multi-dimensional filtering and export, custom PoC combination, and global fingerprint retrieval. <br><img src="./img/task-new1.png" alt="Task New" width="800"><br> <br><img src="./img/task-management.png" alt="Task Management" width="800"><br> * **Threat Intelligence Radar**: Support latest CVE vulnerability tracking and **GitHub** code leakage real-time monitoring. <br><img src="./img/threat-intel-radar1.png" alt="Threat Intelligence Radar" width="800"><br> * **System Settings**: Integrate API hot configuration, dictionary cloud management, queue concurrent hot scaling, and six alarm channels for one-click testing. <br><img src="./img/system-settings1.png" alt="System Settings" width="800"><br> --- ## 🏗️ Architecture Design ARL-Next adopts a microservice architecture with front-end and back-end separation. The core modules are: 1. 🖥️ **Frontend**: Built with **Vue 3.5** + **Vite 5.4**, hosted by **Nginx** in production environment, providing HTTPS security gateway and **Basic Auth Pre-placement Defense**. 2. ⚙️ **Backend**: Based on **Python 3.8+** and **Flask**, handling core business logic and JWT authentication. 3. 🤖 **AI Extension (MCP Server)**: *(New!)* Independently integrate **Model Context Protocol** service, enabling external AI large models/Agent to directly access and schedule underlying retrieval tools. 4. ⚡ **Message and Execution (Broker & Workers)**: Adopt **RabbitMQ** + **Celery** distributed cluster, efficiently decouple and schedule **Nuclei** scanning and threat monitoring and other high-concurrency tasks. 5. 🗄️ **Database**: Based on **MongoDB**, bearing tens of millions of asset data and vulnerability results. 6. 🧩 **Microservices**: *(New!)* Include Node.js Puppeteer rendering container and OSINT intelligence container, specializing in headless rendering and asynchronous information collection, thoroughly eliminating main node task blocking. --- ## 🚀 Deployment Guide ### Production Deployment (Public Cloud One-Click Deployment) ⭐ Recommended **Applicable Scenario**: Domestic cloud server, enterprise intranet. **Benchmark Time**: From scratch deployment takes about **13 minutes**; with Docker environment, it takes only **2 minutes**. **Core Advantages**: * **⚡ Domestic Full-speed**: Direct connection to Alibaba Cloud public image, no fear of GitHub network blocking. * **📦 Lightweight**: No environment configuration, no `docker login`, eliminate redundant compilation chain, and reduce image size by over 700MB. * **🛡️ Extreme Protection**: Automatically issue SSL and force generation of **Basic Auth Pre-placement Interception**, core components fully isolated in the internal network. * **🔄 Smooth Hot Update**: Support one-click smooth restart and upgrade, completely free from SSH login. * **🩺 Intelligent Readiness Detection**: Built-in API health polling mechanism to ensure 100% service startup and seamless access. #### 🚀 Deployment Method Selection Please select the deployment method according to the network environment: **Method 1: One-click Deployment (⭐ Recommended, suitable for domestic servers)** Direct connection in China. Execute directly on a new Ubuntu/Debian (need root), automatically complete dependency installation and service startup: ```bash apt-get update && apt-get install -y docker.io docker-compose-v2 openssl curl && \ mkdir -p ~/ARL-Next && cd ~/ARL-Next && \ docker pull crpi-laul1izptqrf0tkf.cn-beijing.personal.cr.aliyuncs.com/owl234-arl-prod/arl-web:latest && \ docker rm -f arl-temp 2>/dev/null || true && \ docker create --name arl-temp crpi-laul1izptqrf0tkf.cn-beijing.personal.cr.aliyuncs.com/owl234-arl-prod/arl-web:latest && \ docker cp arl-temp:/code/start-prod.sh ./ && \ docker cp arl-temp:/code/docker-compose.prod.yml ./ && \ docker cp arl-temp:/code/updater ./ && \ docker cp arl-temp:/code/version.txt ./ && \ docker rm arl-temp && \ chmod +x start-prod.sh && \ bash start-prod.sh ``` **Method 2: Github Shallow Clone Deployment (suitable for overseas servers / need to keep source code)** > ⚠️ **Note**: This method is highly dependent on the server's network connectivity to Github. If `git clone` is slow or stuck, domestic servers are strongly recommended to use **Method 1 (script extreme speed deployment)**, or manually upload the source code to the server for decompression and operation. If the network allows direct connection to Github, you can directly shallow pull the source code to run (need to install Docker): ```bash git clone --depth 1 https://github.com/owl234/ARL-Next.git && cd ARL-Next chmod +x start-prod.sh bash start-prod.sh ``` Access `https://<your server IP>:5173` to log in. > 🛡️ **Security Login**: > 1. **First Layer Defense Popup (Basic Auth)**: Enter account `admin` / password `arl_next` > 2. **System Login Panel**: Enter account `admin` / password `arlpass` > *(Ignore browser insecurity prompts for the first self-signed certificate)* > ⚙️ **Commercial Certificate Replacement (Optional)**: Rename the certificate to `arl.crt` and `arl.key` and place it in the `ssl-certs/` directory, then re-execute `start-prod.sh`. --- ### Development Environment Deployment (Front-end Local + Docker Back-end) * 👥 **Applicable Group**: Secondary development and security researchers. * ⚡ **Core Advantages**: Front-end and back-end complete decoupling, dual-end hot reload takes effect immediately. * ⚙️ **Preconditions**: Local installation of Docker and Node.js (script will automatically handle pnpm dependencies). #### 🚀 One-click Startup ```bash git clone https://github.com/owl234/ARL-Next && cd ARL-Next bash start-dev.sh # Automatically pull the back-end Docker in the background and start the Vite front-end service in the foreground ``` Access `http://localhost:5173` to start development (default credentials: `admin` / `arlpass`). > 💡 **Development Notes**: > * **Dual-end Hot Reload**: Back-end modification of local code takes effect immediately, front-end Vite real-time hot update. > * **API and Security**: Back-end interface exposure on port `5001`; if you need HTTPS, place the certificate in the `certs/` directory. #### 🛠️ Common Commands ```bash # View status / real-time logs / stop development environment docker compose -f docker-compose.dev.yml ps docker compose -f docker-compose.dev.yml logs -f docker compose -f docker-compose.dev.yml down ``` --- ## 🗄️ Direct Database Connection (Development Environment Exclusive) Production environment defaults to cutting off bottom-level port mapping. Development can be connected directly: **🍃 MongoDB** * **URI**: `mongodb://admin:admin@127.0.0.1:27017/arl?authSource=admin` (account and password: admin) **🐇 RabbitMQ Message Queue** * **AMQP Port**: `5673` * **Web Background**: `http://127.0.0.1:15673` (account and password: `admin` / `admin`) --- ## 📜 Version Update History <details open> <summary><b>v1.1.6 (Current Version)</b></summary><br/> * **Architecture**: Puppeteer is completely separated from the back-end Worker into an independent Node.js HTTP microservice container, greatly releasing back-end scheduling pressure. * **Performance**: Reconstruction of fingerprint recognition engine, introduction of Aho-Corasick multi-mode matching algorithm and memory caching, greatly improving web asset scanning efficiency. * **Crawler**: Optimize URL de-duplication algorithm, introduce Set structure in the bottom layer hash pool to replace List, reducing retrieval complexity from O(N²) to O(1), eliminating CPU bottleneck during large-scale crawling. * **Deployment**: Support Github shallow clone deployment compatibility; start script adds dynamic health detection of API, completely eliminating early 502 errors. * **Fix**: Fixed task list (Task List) and asset detection (Asset Recon) data display anomalies and status synchronization issues. </details> <details> <summary><b>v1.1.5</b></summary><br/> * **Architecture**: Refactor `icp_query` into independent `osint_service` microservice, introduce pure asynchronous scheduling, and reduce main node load. * **Scheduling**: Realize lightweight task queue separation (FOFA and other light queries are independent), and support refined concurrency configuration in the system settings. * **Deployment**: Automatically allocate 2G Swap to solve OOM crash; multi-stage construction reduces image volume; add Autoheal container self-healing service, automatically monitor and restart dead nodes. * **Security**: Hot update service (`updater.py`) adds internal network whitelist interception mechanism, blocking public network calls; fix Nginx and SSE cross-domain restrictions. * **Function**: Task list adds "fuzzy/precise/numerical" condition filtering and combined export; core task layer increases site anti-duplicate insertion mechanism. </details> <details> <summary><b>v1.1.4</b></summary><br/> * **Fix**: Complete missing host collision configuration in policy to ensure back-end tasks can be linked and issued normally. * **Fix**: Fix global background style, solve long page scrolling may appear white edge and background flicker. * **Deployment**: Full refactoring of one-click deployment and hot update underlying robustness. Add concurrent conflict lock, configuration file atomic write, network disconnection automatic retry mechanism; automatically clean up legacy ghost containers and disk discarded images; increase smooth shutdown time (60 seconds) to prevent scan dirty data; and fix multiple edge hidden dangers that may cause deployment paralysis. * **Build**: Upgrade GitHub Actions build dependency version. </details> <details> <summary><b>v1.1.3</b></summary><br/> * **AI Native**: First introduce MCP (Model Context Protocol) Server, enabling external AI large models to seamlessly take over asset scheduling and retrieval. * **UI Refactor**: Front-end style system complete decoupling and refactoring, add dynamic theme color and custom background, creating a geek-exclusive workstation. * **Security**: Production environment Nginx fully enable Basic Auth forced pre-placement interception, container startup automatically generate strong password credentials, achieve extreme protection. * **Function**: Add global asset fingerprint fine-grained retrieval function, support throughout the system to penetrate positioning target site. </details> <details> <summary><b>v1.1.2</b></summary><br/> * **Core**: Add system one-click upgrade mechanism, support smooth hot update. * **Component**: Nuclei scan engine upgraded to v3.11.0. * **Front-end**: Extreme performance optimization, fix Auth interceptor and other verification issues. </details> <details> <summary><b>v1.1.1</b></summary><br/> * **Asset**: Asset range (Scope) expansion, full support and strict distinction between Domain and IP type target verification and scheduling. * **Function**: Add custom PoC source code online reading, editing and full visual creation and management, enhance cascade deletion logic. * **Function**: Add dictionary configuration module, provide weak password dictionary query, preview and visual read and write management. * **Optimization**: 360 search engine collection logic adds anti-crawler fuse protection, append high-value keyword depth mining; production environment Nginx enable Gzip compression. * **Fix**: Fix front-end details page advanced search form and component data linkage anomaly. </details> <details> <summary><b>v1.1.0</b></summary><br/> * **New**: Fully introduce GitHub threat intelligence radar (support CVE vulnerability radar, security weapon library and hacker dynamic monitoring). * **New**: Perfect alarm ecology, support Telegram robot push alarm. * **Refactor**: Front-end system settings and Github management page structure refactoring, fully enable HTTP/2 multiplexing, greatly reduce front-end concurrent loading delay. * **Fix**: Fix HTTP survival detection and site screenshot component in Docker timeout and crash Bug, and fix vulnerability trend data Bug. </details> <details> <summary><b>v1.0.9</b></summary><br/> * **Refactor**: Separate back-end ARL internal vulnerability and Nuclei engine scan result statistics logic. * **Interaction**: Dashboard vulnerability statistics card UI refactoring, support by vulnerability type and hazard level click drill-down. * **Interaction**: Asset query page support receive dashboard linkage request, realize page jump and advanced option automatic filling. </details> <details> <summary><b>v1.0.8</b></summary><br/> * **Function**: Complete POC import mechanism, support batch drag and drop upload verification script, and provide standard Python POC template download. * **Architecture**: Introduce Celery task concurrent hot expansion mechanism, modify concurrency configuration and take effect immediately, no need to restart service. * **Refactor**: Refactor dashboard bottom layer query logic, unified based on site form library for efficient query of massive data. * **Deployment**: Deep separation of development and production environment start script, add POC independent data volume mount. * **Optimization**: Optimize front-end site screenshot preview style anti-deformation, and continuously iterate system built-in fingerprint library. </details> --- ## 🤝 Acknowledgements This project stands on the shoulders of giants, and hereby thanks the following projects and teams: * **Core Architecture**: Based on the original [ARL Beacon](https://github.com/TophantTechnology/ARL) refactoring, and refer to [Aabyss-Team/ARL](https://github.com/Aabyss-Team/ARL) and [adysec/ARL](https://github.com/adysec/ARL) and other excellent derivative versions. * **Fingerprint Engine**: Thanks to **Weiling Security Team** (<img src="img/weiling.jpg" width="18" height="18" align="absmiddle" /> public number) for providing tens of millions of high-quality fingerprint data support. * **Function Module**: Enterprise asset query deeply borrowed from [ICP_Query](https://github.com/HG-ha/ICP_Query), threat monitoring module draws lessons from [github-cve-monitor](https://github.com/yhy0/github-cve-monitor) design ideas. ARL-Next will adhere to the open source and mutual assistance, and continue to contribute to the information security community! --- ## 💖 Sponsorship and Support The continuous development and maintenance of this project cannot be separated from the support of the community. Thank you for the following sponsors' contributions to the open source ecosystem. <p> <img src="./img/buymeacoffee.png" width="200" alt="Buy Me A Coffee" </p> <a href="https://github.com/robotfish001"> <img src="https://github.com/robotfish001.png" width="50" height="50" style="border-radius: 50%" alt="robotfish-001" title="robotfish-001"> </a> <a href="https://github.com/phpmac"> <img src="https://github.com/phpmac.png" width="50" height="50" style="border-radius: 50%" alt="phpmac" title="phpmac"> </a> --- ## ⚠️ Disclaimer and Liability This tool is only for legal authorized enterprise security construction, SRC vulnerability mining and academic research. When using this tool, please make sure to comply with local laws and regulations (such as the <a href="#">Cybersecurity Law of the People's Republic of China</a>) and the testing specifications of the target platform. **Unauthorized probing is an illegal act.** The user shall bear any direct or indirect legal liability arising from the use of this tool, and the project authors and contributors shall not bear any joint liability. ## 💬 Issue Feedback and Communication - **Feedback and Suggestions**: If you encounter any bugs or have functional suggestions, feel free to submit GitHub Issues. - **Technical Exchange**: Welcome to add personal WeChat or join QQ groups to discuss security development and red-blue confrontation technology. - **Get Updates**: Follow the WeChat public account [owl security] to get security dry goods and project update reminders irregularly! <table align="center"> <tr> <td align="center" style="padding: 0 40px;"><b>Personal WeChat</b></td> <td align="center" style="padding: 0 40px;"><b>WeChat Public Account (owl security)</b></td> <td align="center" style="padding: 0 40px;"><b>QQ Exchange Group</b></td> </tr> <tr> <td align="center" style="padding: 0 40px;"><img src="./img/wechat.png" alt="Personal WeChat" width="250" /></td> <td align="center" style="padding: 0 40px;"><img src="./img/wechat_public.jpg" alt="WeChat Public Account owl security" width="250" /></td> <td align="center" style="padding: 0 40px;"><img src="./img/qq_group.jpg" alt="QQ Exchange Group" width="250" /></td> </tr> </table> ## 🌟 Star History **⭐ If this project brings convenience to your security work, give it a Star for support!** <div align="center"> <a href="https://www.star-history.com/?repos=owl234%2Farl-next&type=date&legend=top-left"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=owl234/arl-next&type=date&theme=dark&legend=top-left&sealed_token=vNF3XBBUYjnOkZ1XfTODaJEURB73qlNr1zXyCH6HOUbJGKju3QmIb7pVDyjCK67Ra-ukzG7dgZ3B3HDpCKJ3raveN9bOCec7r6gDILhjGrYbcVEV2Gy5Ew" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=owl234/arl-next&type=date&legend=top-left&sealed_token=vNF3XBBUYjnOkZ1XfTODaJEURB73qlNr1zXyCH6HOUbJGKju3QmIb7pVDyjCK67Ra-ukzG7dgZ3B3HDpCKJ3raveN9bOCec7r6gDILhjGrYbcVEV2Gy5Ew" /> <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=owl234/arl-next&type=date&legend=top-left&sealed_token=vNF3XBBUYjnOkZ1XfTODaJEURB73qlNr1zXyCH6HOUbJGKju3QmIb7pVDyjCK67Ra-ukzG7dgZ3B3HDpCKJ3raveN9bOCec7r6gDILhjGrYbcVEV2Gy5Ew" /> </picture> </a> </div>