FoundryBrowserAutomation

# 🚀 Foundry Local & Browser Automation Demo **A modern Electron desktop application that demonstrates the integration of Microsoft Foundry Local SDK for AI model inference with Playwright MCP (Model Context Protocol) for intelligent browser automation.**     ## ✨ What This App Does This application showcases the powerful combination of local AI models and browser automation in a sleek, modern interface. It demonstrates how AI can intelligently control web browsers using natural language commands, making web automation accessible to both developers and non-technical users. ### 🎯 Key Capabilities - **🤖 Local AI Integration**: Execute AI models locally using Microsoft Foundry Local SDK - **🌐 Intelligent Browser Automation**: Control browsers with natural language using Playwright MCP - **💬 Natural Language Commands**: Tell the browser what to do in plain English - **📸 Visual Feedback**: Automatic screenshot capture and display - **🎨 Modern UI**: Beautiful tabbed interface with dark/light mode support - **📊 Real-time Monitoring**: Live logs and status indicators - **🔧 Built-in Diagnostics**: Comprehensive testing and troubleshooting tools ## 🖼️ Application Interface The application features a modern, tabbed interface with three main sections: 1. **AI Model Tab**: Configure and test AI model connections 2. **Browser Automation Tab**: Execute natural language browser commands 3. **Logs Tab**: Monitor real-time application activity *Note: To see the application interface, run the app using the instructions below.* ## 🚀 Quick Start ### Prerequisites - **Node.js v16+** and npm - **Microsoft Foundry Local SDK** (optional - app includes fallback functionality) - **Windows/macOS/Linux** (cross-platform support) ### Installation & Setup 1. **Clone the repository** ```bash git clone <repository-url> cd electron-app ``` 2. **Install dependencies** (automatically installs Playwright browsers) ```bash npm install ``` 3. **Start the application** ```bash npm start ``` That's it! The app will automatically set up everything needed and launch with a beautiful splash screen. ## 📋 How to Use ### 🤖 AI Model Testing 1. Navigate to the **AI Model** tab 2. Select a model from the dropdown (if Foundry Local is available) 3. Enter a prompt and click **Submit** 4. View the AI response in the interface ### 🌐 Browser Automation 1. Go to the **Browser Automation** tab 2. Try these example commands in the MCP prompt field: **Basic Navigation:** ``` Go to microsoft.com and take a screenshot ``` **Complex Interactions:** ``` Navigate to github.com, search for "playwright", and screenshot the results ``` **Content Extraction:** ``` Visit wikipedia.org, search for "artificial intelligence", and take a screenshot of the article ``` 3. Click **Run MCP Prompt** to execute 4. View screenshots and results in the interface ### 🔧 Testing & Diagnostics - **Run MCP Test**: Click to run comprehensive browser automation tests - **Diagnose MCP**: Check MCP installation and configuration - **View Logs**: Monitor all application activity in real-time ## 🛠️ Available Commands | Command | Description | |---------|-------------| | `npm start` | Launch the application with auto-setup | | `npm test` | Run all automated tests | | `npm run test:mcp` | Test Playwright MCP functionality | | `npm run diagnose:mcp` | Diagnose MCP installation issues | | `npm run fix:mcp` | Reinstall MCP components | | `npm run package` | Build distributable packages | ## 🏗️ Architecture ### Core Components ``` ├── main.js # Electron main process & IPC handling ├── renderer.js # UI logic & event handling ├── preload.js # Secure IPC bridge ├── browser-automation.js # Playwright MCP integration ├── fallback-mcp.js # Fallback automation (when MCP unavailable) ├── theme.js # Dark/light mode & UI animations └── tests/ ├── mcp-test.js # Comprehensive MCP testing └── find-mcp.js # MCP diagnostics ``` ### UI Structure ``` ├── index.html # Main application layout ├── styles.css # Modern styling & animations ├── animation.css # UI transition effects └── splash.html # Startup splash screen ``` ## 🎨 Features Showcase ### 🌓 Theming - **Automatic dark/light mode** based on system preferences - **Manual theme toggle** with smooth transitions - **Consistent color scheme** throughout the interface ### 📱 Responsive Design - **Tabbed interface** for organized content - **Card-based layout** with subtle shadows and rounded corners - **Responsive controls** that adapt to window size ### ⚡ Performance - **Lazy loading** of heavy components - **Efficient IPC communication** between processes - **Background processing** for non-blocking operations ### 🔒 Security - **Context isolation** enabled in Electron - **Secure IPC** communication via preload scripts - **Sandboxed renderer** process ## 🧪 Testing ### Automated Tests **Run all tests:** ```bash npm test ``` **Test specific functionality:** ```bash npm run test:mcp # Test browser automation npm run test:browser # Test basic browser functionality ``` ### Manual Testing Examples Try these commands to test the application: 1. **Simple Navigation Test** ``` Go to example.com and take a screenshot ``` 2. **Search Test** ``` Navigate to duckduckgo.com, search for "electron apps", take a screenshot ``` 3. **Multi-step Test** ``` Go to github.com, click on "Explore", then take a screenshot of the page ``` ## 🚨 Troubleshooting ### Common Issues **MCP Not Working?** ```bash npm run diagnose:mcp # Check MCP installation npm run fix:mcp # Reinstall MCP components ``` **Browser Won't Launch?** ```bash npx playwright install # Install browser binaries ``` **Connection Issues?** - Ensure Foundry Local is running (if using AI features) - Check firewall settings - Verify network connectivity ### Getting Help 1. **Check the Logs tab** in the application for detailed error messages 2. **Run diagnostics** using `npm run diagnose:mcp` 3. **Check the console** output when running from terminal 4. **Review error messages** in the application interface ## 🔧 Advanced Configuration ### Custom Model Configuration Edit the model dropdown options in `renderer.js` to add your preferred models. ### Browser Automation Settings Modify browser launch options in `browser-automation.js` for specific requirements. ### UI Customization Update `styles.css` and `animation.css` to customize the application appearance. ## 📦 Building & Distribution **Create distributable packages:** ```bash npm run package ``` This creates platform-specific packages in the `dist/` directory: - **Windows**: `.exe` installer - **macOS**: `.dmg` disk image - **Linux**: `.AppImage` portable app ## 🛡️ Security & Privacy - **Local Processing**: All AI inference happens locally (no cloud dependency) - **Secure Automation**: Browser automation runs in isolated contexts - **No Data Collection**: Application doesn't collect or transmit user data - **Open Source**: Full source code available for review ## 📚 Technologies Used | Technology | Purpose | Version | |------------|---------|---------| | **Electron** | Desktop app framework | ^29.0.0 | | **Playwright** | Browser automation | ^1.52.0 | | **@playwright/mcp** | Natural language browser control | ^0.0.26 | | **Node.js** | Runtime environment | 16+ | | **Microsoft Foundry Local** | Local AI inference | Latest | ## 🎯 Use Cases This application demonstrates several practical use cases: - **🧪 AI Model Testing**: Quick testing of local AI models - **🕷️ Web Scraping**: Automated data extraction from websites - **🧑‍💻 Browser Testing**: Automated UI testing with natural language - **📊 Monitoring**: Automated website monitoring and screenshot capture - **🎓 Education**: Learning browser automation and AI integration - **🚀 Prototyping**: Rapid prototyping of automation workflows ## 🔮 Future Enhancements - **🤖 Advanced AI Integration**: Support for more AI providers - **📝 Script Recording**: Record browser actions as reusable scripts - **🔄 Workflow Builder**: Visual workflow creation interface - **☁️ Cloud Integration**: Optional cloud AI model support - **📱 Mobile Support**: Electron-based mobile app version ## 📄 License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ## 🤝 Contributing Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change. ## 🆘 Support If you encounter any issues or have questions: 1. **Check the troubleshooting section** above 2. **Run the built-in diagnostics** (`npm run diagnose:mcp`) 3. **Review the application logs** in the Logs tab 4. **Open an issue** on GitHub with detailed information --- **Built with ❤️ using Electron, Playwright, and Microsoft Foundry Local SDK**