RAG Challenge Technical Comparison:
A Reproducible Post-Mortem

What This Post Is About

I participated in a RAG (Retrieval-Augmented Generation) challenge organized by ARSII in partnership with Rose Blanche Group. The challenge had clear, written rules specifying exact technical constraints. The winning solution does not use PostgreSQL, which the competition PDF specifies as a mandatory requirement. I raised this with the organizers. No response has been received as of the publication date.

This post presents the evidence — the competition rules, the code, and reproducible benchmark results — so that anyone can verify the claims independently.

This analysis focuses on the technical artifacts — code, configurations, and measurable retrieval outputs — not on any individual participant.

The Competition Rules

The challenge was proposed by Rose Blanche Group (STE AGRO MELANGE TECHNOLOGIE). The prize: a 1000 DT gift voucher.

Official Challenge Description

Développer un module de recherche sémantique permettant d'interroger cette base vectorielle (RAG)

EN: Develop a semantic search module to query this vector database (RAG)

Dans un contexte où une base documentaire contient un grand volume d'informations [...] L'objectif est de développer un module intelligent capable d'assister l'utilisateur en retrouvant automatiquement les fragments les plus pertinents à partir d'une question formulée en langage naturel.

EN: In a context where a document base contains a large volume of information [...] The objective is to develop an intelligent module capable of assisting the user by automatically retrieving the most relevant fragments from a question formulated in natural language.

Official Technical PDF — Exact Constraints

The competition PDF (Développement d'un Module de Recherche Sémantique pour la Formulation en Boulangerie & Pâtisserie) specifies verbatim:

Context: Ces fiches ont déjà été : Converties en texte, Découpées en fragments (chunks), Transformées en embeddings, Stockées dans une base PostgreSQL

EN: These data sheets have already been: Converted to text, Split into fragments (chunks), Transformed into embeddings, Stored in a PostgreSQL database

Resources provided: Base de données PostgreSQL — Table : embeddings — Structure : id (Primary Key), id_document (int), texte_fragment (text), vecteur (VECTOR(384))

EN: PostgreSQL database — Table: embeddings — id (Primary Key), id_document (int), texte_fragment (text), vecteur (VECTOR(384))

Imposed parameters: Méthode de similarité : Cosine Similarity — Nombre de résultats à retourner : Top K = 3 — Langage recommandé : Python

Required embedding model: Modèle imposé : all-MiniLM-L6-v2 — Bibliothèque : sentence-transformers — Dimension : 384

Example query: Améliorant de panification : quelles sont les quantités recommandées d'alpha-amylase, xylanase et d'Acide ascorbique ?

EN: Bread improver: what are the recommended quantities of alpha-amylase, xylanase, and ascorbic acid?

The database constraint is not a suggestion. The PDF says the data is "Stockées dans une base PostgreSQL" and provides an exact table schema with typed columns including vecteur VECTOR(384) — a data type that only exists in PostgreSQL via the pgvector extension. The entire challenge is framed around querying this PostgreSQL vector database.

What This Challenge Is Really About

The constraints above are not arbitrary. They define a very specific engineering problem: given a small embedding model (22M parameters, 384 dimensions), a fixed retrieval depth (K=3), and a local PostgreSQL database — how well can you preprocess, structure, and optimize your data to produce the best possible retrieval results?

This is a data engineering and optimization challenge, not an API integration exercise. The fixed model, fixed K, and local database requirement exist precisely to level the playing field and measure what each participant does with the data itself — the PDF extraction quality, the chunking strategy, the way embeddings are structured, and how queries are handled.

If the challenge were simply about building a RAG system, anyone could plug in LangChain with a powerful LLM API and get fluent answers in minutes. But that approach requires no optimization, incurs high API costs, sends potentially sensitive enterprise data to external providers, and demonstrates no understanding of the underlying retrieval mechanics. The competition deliberately eliminates that shortcut by imposing a local, self-contained setup.

The real skill being tested is: can you make a 22-million-parameter model punch above its weight through intelligent data preprocessing and search engineering?

The Winning Solution Does Not Use PostgreSQL

The winning team's code is publicly available at github.com/AyaZantour/RagProject. Here is what their storage layer actually does.

vector_store.py — Lines 57–68 (Save method)

def save(self, path: str) -> None:
    os.makedirs(os.path.dirname(path) if os.path.dirname(path) else ".", exist_ok=True)
 
    np.savez_compressed(f"{path}.npz", embeddings=self.embeddings)
    with open(f"{path}.json", "w", encoding="utf-8") as f:
        json.dump(self.metadata, f, ensure_ascii=False, indent=2)

vector_store.py — Lines 94–119 (Search method)

def search(self, query: str, top_k: int = 3) -> list:
    # Embed the query
    query_embedding = self.model.encode(
        [query],
        convert_to_numpy=True,
        normalize_embeddings=True
    )
 
    # Cosine similarity (dot product since vectors are normalized)
    similarities = np.dot(self.embeddings, query_embedding.T).flatten()
 
    # Get top-k indices
    top_indices = np.argsort(similarities)[::-1][:top_k]

Search is performed via NumPy dot product on in-memory arrays. No database query. No SQL ORDER BY. No <=> pgvector operator.

requirements.txt

PyPDF2>=3.0.0
sentence-transformers>=2.2.0
numpy>=1.24.0
flask>=3.0.0
groq>=0.4.0

Required table schema (from competition PDF)

CREATE TABLE embeddings (
    id             SERIAL PRIMARY KEY,
    id_document    INT,
    texte_fragment TEXT,
    vecteur        VECTOR(384)
);

The VECTOR(384) type only exists in PostgreSQL via the pgvector extension. The PDF does not name pgvector explicitly, but the schema necessarily implies it. Either way, this is a PostgreSQL table — not NumPy files.

My Solution — Full Compliance

My code is publicly available at github.com/mohamedyaakoubi/RBG-ARSII-RAG.

docker-compose.yml — PostgreSQL with pgvector

services:
  postgres-vector:
    build: .
    container_name: postgres-vector-db
    environment:
      POSTGRES_USER: rag
      POSTGRES_PASSWORD: ragpassword
      POSTGRES_DB: ragdb
    ports:
      - "5432:5432"

Dockerfile — PostgreSQL 16 + pgvector

FROM postgres:16
 
RUN apt-get update && apt-get install -y \
    git build-essential postgresql-server-dev-16
 
RUN git clone https://github.com/pgvector/pgvector.git \
    && cd pgvector && make && make install
 
COPY init.sql /docker-entrypoint-initdb.d/

init.sql

CREATE EXTENSION IF NOT EXISTS vector;

Exact required table schema (database/models.py)

cursor.execute(f"""
    CREATE TABLE IF NOT EXISTS embeddings (
        id              SERIAL PRIMARY KEY,
        id_document     INT,
        texte_fragment  TEXT,
        vecteur         vector({config.EMBEDDING_DIMENSION})
    )
""")

Cosine similarity search via pgvector (database/models.py)

cursor.execute("""
    SELECT
        id_document,
        texte_fragment,
        1 - (vecteur <=> %s::vector) AS score
    FROM embeddings
    ORDER BY vecteur <=> %s::vector
    LIMIT %s
""", (query_vector.tolist(), query_vector.tolist(), top_k))

The <=> operator is the pgvector cosine distance operator — consistent with the challenge's VECTOR(384) column type and cosine similarity requirement.

Constraint Compliance Summary

Retrieval Quality — Reproducible Benchmark

Beyond compliance, the core purpose of the challenge is retrieval quality. Both solutions were tested with the same 16 queries, same embedding model (all-MiniLM-L6-v2), same K=3.

How to Reproduce

Winning solution:

• Clone github.com/AyaZantour/RagProject
• pip install -r requirements.txt
• Run python app.py — the index auto-builds on first run
• POST to /search with {"query": "your question"} or use the web UI

My solution:

• Clone github.com/mohamedyaakoubi/RBG-ARSII-RAG
• docker-compose up -d to start PostgreSQL
• pip install -r requirements.txt
• python main.py (runs ingestion + starts Streamlit UI)

Benchmark Results

Score = cosine similarity of the top-1 retrieved chunk. Higher = better retrieval.

Answer Helpfulness — What Each System Actually Returns

Beyond similarity scores, the critical question is: does the retrieved fragment actually answer the user's question?

6.1 Competition Example Query

6.2 French Queries — Language Handling

6.3 Specific Product Queries

6.4 Helpfulness Assessment Summary

Chunk Quality Comparison

My solution — Entity-centric chunks (pdfplumber extraction)

Dosage alpha-amylase (BVZyme AF330) boulangerie panification : 2-10.

BVZyme HCB710 (xylanase): Product: Enzyme preparation based on endo-xylanase.
Source: Bacterial xylanase produced by fermenting a selected unique strain of
Bacillus subtilis. Activity: 583 XylH/g.

BVZyme AF110 (alpha-amylase) storage conditions and shelf life: minimum
durability: 24 months. Store in a cool, dry place (below 20°C).

Winning solution — Fixed-width sliding window (PyPDF2 extraction)

Product Description
Enzyme preparation based on Maltogenic Amylase
 
 
 Effective material
 
 
 
 
 Improve freshness, enhance softness, and extend shelf life.
 
 
 Dosage
Aspect: free flowing powder
Color: white -cream
 Physicochemical
Moisture: <15%

ustries.TECH NICAL DATA SHEET   BVZyme TG MAX64 « Bakery Enzyme
Product Description     BVZyme  TG MAX64 « is used in bakery as a strong
protein cross-linking(connecting residues of the amino acid L-glutamine
to the amino acid £¿L-lysine)Application

Why the Technical Gap Exists

This is not about one team being "better" — it is about the architectural choices that the challenge was designed to test.

PDF Extraction

Chunking Strategy

Embedding Strategy

Search Engineering

Scope and Approach — What Each Solution Prioritized

The LLM has no access to the original PDFs or any external knowledge source — it only sees the 3 retrieved chunks. This means the retrieval errors documented in Section 6 propagate directly into the final answer. When retrieval returns incorrect content, the LLM-generated answer inherits those errors regardless of how fluent the output appears.

Scalability — How Each System Handles New Products

Scalability was not a stated competition requirement. But a RAG system that only works on the data it was built for is not a RAG system — it's a lookup table. The whole point of retrieval-augmented generation is that new documents can be added and queried without modifying the pipeline. This section tests retrieval generalization: whether each system's engineering approach is genuinely robust or merely fitted to the original 35 PDFs.

To test how each solution handles growth, I added 6 new product PDFs that were never in the original dataset: GEbake Amyl, Domax SF Bingo Plus, Tigris Gold, O-TENTIC DURUM, an ascorbic acid spec sheet, and an additives authorization document.

10 queries were run against these new products. Each result was evaluated not by cosine score alone, but by answer helpfulness: does R1 come from the correct product PDF, and does the returned fragment actually contain the answer? Specific values were cross-verified against the source PDFs using pdfplumber.

The entity-centric chunking strategy described in Section 7 prefixes each fragment with the product name, allowing the embedding model to distinguish products by identity even as the corpus grows. Without that anchoring, the winning solution's generic-header chunks compete across PDFs — returning acide ascorbique fragments for Tigris Gold and O-TENTIC DURUM queries.

Storage vs Accuracy — The Space-Compute Tradeoff

As noted in Section 10, scalability is not a scored metric. This section examines the storage and compute cost of each solution's architectural choices.

My solution stores 9.3× more embeddings than the winning solution (1,635 vs 176). The natural question is: what does that cost, and what does it buy?

The competition imposes a 22-million-parameter model with 384 dimensions — deliberately small and cheap. With a fixed model, the only optimization lever is how you prepare the data for it. Entity-centric chunking produces more vectors, but each one is semantically focused: one topic, one product, one answerable question. This gives the small model a much higher probability of producing a strong cosine match, because the 384 dimensions encode a single concept rather than a mixture of unrelated fields.

The winning solution takes the opposite path: fewer, larger chunks that mix product descriptions, dosage tables, physical properties, and contact information into single 500-character windows. This saves negligible storage but forces the embedding model to compress multiple unrelated concepts into a single 384-dimensional vector — exactly the scenario where a small model loses discriminative power between topics.

In information retrieval theory, this is the precision-recall tradeoff: more, smaller chunks increase recall (more potential matches) at the cost of storage overhead. For a corpus of 35 PDFs, that overhead is 2.24 MB. For context, the winning solution's per-query external LLM API call introduces more latency and operational cost than the entire additional storage.

The tradeoff is clear: spend 2.24 MB of disk space and a few extra milliseconds of compute per query, or accept 40.4% lower retrieval accuracy. Both are valid architectural choices — but in a challenge designed to test retrieval quality within tight constraints, one of them is better aligned with the problem.

How Storage Scales With New Products

Rose Blanche Group's enzyme catalog will grow over time. How does each approach scale?

At 35 PDFs, noobmaster averages ~46.7 embeddings per PDF (entity-centric: 9 structured categories × ~5.2 products per PDF, plus enrichment variants). The winner averages ~5 chunks per PDF (500-char windows over short TDS documents).

Search latency tells a more interesting story. noobmaster uses PostgreSQL with pgvector, which supports HNSW and IVFFlat approximate nearest-neighbor indexes. These reduce search from O(n) brute-force to O(log n) at query time, keeping latency sub-10ms even at hundreds of thousands of vectors. The winner uses NumPy brute-force (np.dot over the full array) with no indexing structure — at 50,000 vectors this remains fast (~5ms), but the architecture offers no path to sub-linear scaling without a fundamental rewrite.

The real cost asymmetry is elsewhere: the winner's Groq API call per query has variable latency (network-dependent), a per-token monetary cost, and a hard dependency on an external service being available. My solution's additional storage costs nothing to operate once written to disk.

The Organizational Response

On March 18, 2026, I sent a formal email to ARSII with the full technical comparison report, requesting an explanation of the evaluation criteria used.

ARSII's response was a broadcast email to all participants (not a direct reply). The relevant points:

"We would like to clarify that ARSII does not interfere in the evaluation process of the challenges, except for those proposed by ARSII itself. Each challenge is assessed independently by the respective partner organization responsible for it."

"In case of any objections or concerns, you are absolutely welcome to submit a reclamation. We will ensure that you are put in direct contact with the responsible party of the challenge."

According to this response, ARSII delegates challenge evaluation to the respective partner organization. The recourse offered is direct contact with Rose Blanche Group.

As of March 26, 2026, no direct response has been received and no acknowledgment of the documented requirement discrepancy has been issued.

What This Post Establishes

This post is a public record of findings, not a request. The evidence presented here is independently verifiable by anyone with access to the two GitHub repositories and the competition PDF. Specifically:

• The winning solution does not use PostgreSQL — observable in the source code (Section 2).
• Retrieval quality is measurably lower on every metric tested — cosine similarity, answer helpfulness, and product accuracy.
• The evaluation process has not been explained. What criteria were used? If the PostgreSQL requirement was waived, when and how were participants informed?

These are documented observations. The reader can draw their own conclusions.

All Evidence — Verify It Yourself

Everything mentioned in this post is publicly verifiable:

The data presented in this post is fully reproducible. Anyone can clone both repositories, run the same queries, and independently verify every metric reported here.