OpenDeepResearcher

OpenDeepResearcher: An AI-Powered Tool for In-Depth Research

OpenDeepResearcher is an open-source notebook that functions as an AI researcher. It continuously searches for information based on a user's query until it gathers all necessary details. Leveraging services like SERPAPI, Jina, and OpenRouter, it automates the research process, providing comprehensive reports on various topics. This tool is designed to streamline research, making it more efficient and thorough.

What is OpenDeepResearcher?

OpenDeepResearcher is a Jupyter notebook that implements an AI researcher. It's designed to automatically gather information on a given topic using a combination of search engines, web content extraction, and large language models (LLMs).

How does OpenDeepResearcher work?

The tool operates through an iterative research loop, refining search queries and processing information asynchronously. Here’s a breakdown of its workflow:

1. Input & Query Generation: The user provides a research topic, and the LLM generates up to four distinct search queries.
2. Concurrent Search & Processing:
   • SERPAPI: Each search query is sent to SERPAPI concurrently.
   • Deduplication: All retrieved links are aggregated and deduplicated within the current iteration.
   • Jina & LLM: Each unique link is processed concurrently to fetch webpage content via Jina, evaluate its usefulness with the LLM, and extract relevant information if the page is deemed useful.
3. Iterative Refinement: The system passes the aggregated context to the LLM to determine if further search queries are needed. New queries are generated if required; otherwise, the loop terminates.
4. Final Report Generation: All gathered context is compiled and sent to the LLM to produce a final, comprehensive report addressing the original query.

Key Features:

• Iterative Research Loop: System refines search queries iteratively until no further queries are required.
• Asynchronous Processing: Searches, webpage fetching, evaluation, and context extraction are performed concurrently to improve speed.
• Duplicate Filtering: Aggregates and deduplicates links within each round, ensuring that the same link isn’t processed twice.
• LLM-Powered Decision Making: Uses the LLM to generate new search queries, decide on page usefulness, extract relevant context, and produce a final comprehensive report.
• Gradio Interface: Offers a functional UI via the open-deep-researcher - gradio notebook.

How to use OpenDeepResearcher?

1. Setup:
   • Clone or open the notebook in Google Colab.
   • Install the nest_asyncio library.
   • Configure API keys for OpenRouter, SERPAPI, and Jina.
2. Usage:
   • Run the notebook cells in order.
   • Provide a research query/topic.
   • Specify an optional maximum number of iterations.
3. Follow the Research Process: The notebook will generate initial search queries, perform asynchronous searches, and iteratively refine the search until a final report is generated.
4. View the Final Report: The comprehensive report will be printed in the output.

Why choose OpenDeepResearcher?

OpenDeepResearcher offers a unique approach to automated research by combining the power of search engines, web content extraction, and LLMs. It's particularly useful for:

• Researchers who need to gather information quickly and efficiently.
• Analysts who need to analyze large amounts of data.
• Anyone who wants to learn more about a particular topic.

Who is OpenDeepResearcher for?

This tool is ideal for:

• Researchers
• Data analysts
• Students
• Anyone who needs to conduct in-depth research on a topic.

Requirements

To use OpenDeepResearcher, you'll need API access and keys for the following services:

• OpenRouter API
• SERPAPI API
• Jina API

Troubleshooting

• RuntimeError with asyncio: Ensure you have applied nest_asyncio as shown in the setup section.
• API Issues: Verify that your API keys are correct and that you are not exceeding any rate limits.

OpenDeepResearcher is released under the MIT License. See the LICENSE file for more details.

By leveraging the power of AI and automation, OpenDeepResearcher makes the research process more efficient and comprehensive. It is a valuable tool for anyone looking to gather in-depth information on a wide range of topics.