Like humans, large language models (LLMs) often have differing skills and strengths derived from differences in their architectures and training regimens. However, they struggle to combine specialized expertise across different domains, limiting their problem-solving capabilities compared to humans. Specialized models like MetaMath, WizardMath, and QwenMath excel at mathematical reasoning but often underperform on tasks requiring common sense or medical knowledge. Even within specific domains such as mathematics, models show nuanced variations in capability, e.g., one might excel at algebra while another masters geometry. creates a need for frameworks that can identify and select the most appropriate expert models for specific problems.

Existing approaches like Mixture-of-Experts (MoE) models distribute computation across multiple specialized components, with recent emphasis on sparse approaches that activate only the most relevant experts per input. The Sparse MoE (SMoE) method has improved efficiency across vision, language, and multimodal tasks but requires combining models in the parameter space through joint training. More recent frameworks like MoA (Mixture-of-Agents) attempt to address this by combining LLM outputs symbolically. Further, Multi-agent reasoning approaches have emerged as alternatives, such as the Student-teacher technique that distills reasoning capabilities from stronger to weaker agents, while debate frameworks allow multiple agents to refine arguments collectively.

Researchers from UNC Chapel Hill have proposed SYMBOLIC-MOE, a symbolic, text-based, and gradient-free Mixture-of-Experts framework to enable adaptive instance-level mixing of pre-trained LLM experts. It takes a fine-grained perspective by emphasizing specialized skills within broader domains like algebra within mathematics or molecular biology within biomedical reasoning. They also introduced a skill-based recruiting strategy that dynamically selects the most relevant expert LLMs for each specific reasoning task based on their demonstrated strengths. Moreover,  SYMBOLIC-MOE outperforms strong LLMs like GPT4o-mini, as well as multiagent approaches, with an absolute average improvement of 8.15% over the best multi-agent baseline.

SYMBOLIC-MOE consists of three stages: model profile creation and aggregator selection followed by expert recruitment and final answer generation, both of which take place during inference. To maximize throughput and efficiency, SYMBOLIC-MOE introduces an innovative batching strategy where all instances are first analyzed to determine which LLMs will be needed. The system then intelligently groups problem instances based on their required experts, allowing each active expert model to receive all relevant instances in a single batch and ensuring each expert is loaded only once. This solution enables efficient batched inference on a single GPU while supporting a diverse pool of 16 LLMs, with the flexibility to add more GPUs for further parallelization.

SYMBOLIC-MOE shows exceptional performance across diverse benchmarks. It consistently outperforms all baseline approaches, surpassing single-model strategies, multi-agent debates with a single model, and multi-model multi-agent frameworks like MoA and ReConcile. It exceeds the strongest multi-agent baseline (Self-MoA) by an impressive 8.15% absolute average improvement, 8.28% on MMLU-Pro, 13.45% on AIME, 4.92% on GPQA, and 6.08% on MedMCQA. SYMBOLIC-MOE achieves comparable or superior performance to larger models with 70B parameters by using four 7-8B parameter models. It outperforms Llama3.3 70B on AIME and GPQA while matching its performance on MedMCQA. Efficiency testing reveals that it operates 44% faster on a single GPU than MoA while achieving better accuracy.

In conclusion, researchers introduced SYMBOLIC-MOE, a scalable MoE framework that combines models through their symbolic output. This method identifies the skills needed for a given problem and recruits agents based on those skills to engage in a discussion about a given input. SYMBOLIC-MOE outperforms standard inference-time scaling methods as well as other debate frameworks and other mixture-of-agents methods, leading to strong performance across domains without human intervention. It’s average performance across heterogeneous tasks is in fact stronger than that of advanced proprietary models such as GPT4o-mini. However, this method has limitations: (a) It involves running multiple models, which increases inference cost, and (b) it relies on skills inferred from a small validation set to set the agent profiles.

---

Check out the Paper and GitHub Page. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 80k+ ML SubReddit.






Sajjad Ansari is a final year undergraduate from IIT Kharagpur. As a Tech enthusiast, he delves into the practical applications of AI with a focus on understanding the impact of AI technologies and their real-world implications. He aims to articulate complex AI concepts in a clear and accessible manner.

Parlant: Build Reliable AI Customer Facing Agents with LLMs 💬 ✅ (Promoted)

In this tutorial, we demonstrate how to build an AI-powered PDF interaction system in Google Colab using Gemini Flash 1.5, PyMuPDF, and the Google Generative AI API. By leveraging these tools, we can seamlessly upload a PDF, extract its text, and interactively ask questions, receiving intelligent responses from Google’s latest Gemini Flash 1.5 model.

!pip install -q -U google-generativeai PyMuPDF python-dotenv

First we install the necessary dependencies for building an AI-powered PDF Q&A system in Google Colab. google-generativeai provides access to Gemini Flash 1.5, enabling natural language interactions, while PyMuPDF (also known as Fitz) allows efficient text extraction from PDFs. Also, python-dotenv helps manage environment variables, such as API keys, securely within the notebook.

from google.colab import files
uploaded = files.upload()

We upload files from your local device to Google Colab. When executed, it opens a file selection dialog, allowing you to choose a file (e.g., a PDF) to upload. The uploaded file is stored in a dictionary-like object (uploaded), where keys represent file names and values contain the file’s binary data. This step is essential for directly processing documents, datasets, or model weights in a Colab environment.

import fitz


def extract_pdf_text(pdf_path):
    doc = fitz.open(pdf_path)
    full_text = ""
    for page in doc:
        full_text += page.get_text()
    return full_text


pdf_file_path="/content/Paper.pdf"
document_text = extract_pdf_text(pdf_path=pdf_file_path)
print("Document text extracted!")
print(document_text[:1000]) 

We use PyMuPDF (fitz) to extract text from a PDF file in Google Colab. The function extract_pdf_text(pdf_path) reads the PDF, iterates through its pages, and retrieves the text content. The extracted text is then stored in document_text, with the first 1000 characters printed to preview the content. This step is crucial for enabling text-based analysis and AI-driven question answering from PDFs.

import os
os.environ["GOOGLE_API_KEY"] = 'Use your own API key here'

We set the Google API key as an environment variable in Google Colab. The API key is required to authenticate requests to Google Generative AI, allowing access to Gemini Flash 1.5 for AI-powered text processing. Replacing ‘Use your own API key here’ with a valid key ensures that the model can generate responses securely within the notebook.

import google.generativeai as genai


genai.configure(api_key=os.environ["GOOGLE_API_KEY"])


model_name = "models/gemini-1.5-flash-001"


def query_gemini_flash(question, context):
    model = genai.GenerativeModel(model_name=model_name)
    prompt = f"""
Context: {context[:20000]}


Question: {question}


Answer:
"""
    response = model.generate_content(prompt)
    return response.text


pdf_text = extract_pdf_text("/content/Paper.pdf")


question = "Summarize the key findings of this document."
answer = query_gemini_flash(question, pdf_text)
print("Gemini Flash Answer:")
print(answer)

Finally, we configure and query Gemini Flash 1.5 using a PDF document for AI-powered text generation. It initializes the genai library with the API key and loads the Gemini Flash 1.5 model (gemini-1.5-flash-001). The query_gemini_flash() function takes a question and extracted PDF text as input, formulates a structured prompt, and retrieves an AI-generated response. This setup enables automated document summarization and intelligent Q&A from PDFs.

In conclusion, following this tutorial, we have successfully built an interactive PDF-based interaction system in Google Colab using Gemini Flash 1.5, PyMuPDF, and the Google Generative AI API. This solution enables users to extract information from PDFs and interactively query them easily. The combination of Google’s cutting-edge AI models and Colab’s cloud-based environment provides a powerful and accessible way to process large documents without requiring heavy computational resources.

Here is the Colab Notebook. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 80k+ ML SubReddit.