ETCC

About this project

The challenge

ETCC's site is the public listing of technology assessment projects its member organizations fund and run, so almost none of the material originates on the site itself. Members publish through their own channels, and keeping a central listing current by hand across many organizations stops working quickly. A report released as a PDF is also published without necessarily being usable: readers have to download it and open it somewhere else, and ordinary site search cannot reach what is inside it, so a question whose answer is on page 40 of the attachment goes unanswered. And the people following this work over time need to hear when something new lands, without revisiting the listing to check.

What we built

We built it as a Drupal application rather than a publishing site. A custom daily import collects project reports from the external sources members publish through, so the central listing assembles itself instead of depending on somebody collating submissions by hand. Visitors can follow individual reports and receive custom email notifications when something they are tracking is published or changes, so the site tells them when there is something new. We extended Drupal's Search API into a lookup built specifically for reports, so a visitor narrows to what they need by the attributes that distinguish one project from another, not by searching page text. And custom JavaScript renders project PDFs as images in the page, so a report can be read where it is found, in any browser, without downloading a file and opening a separate application first. Beyond that, we built a retrieval-augmented generation pipeline so the contents of the PDFs became searchable in their own right. Each document is put through OCR to recover its text and layout, chunked on paragraph and heading boundaries, and embedded as vectors that carry document-level context with every chunk, with the chunks and their full text stored together. A query is embedded the same way and run as a hybrid search across both the vectors and the full text, the two result sets are combined with Reciprocal Rank Fusion, a reranker refines what survives, and only then does the result reach the model that answers the question.

The result

The listing stays current as members publish, with nobody in the middle maintaining it, and anyone following a project hears about it when it changes instead of having to remember to look. A visitor can also ask a question in plain language and get an answer drawn from the body of a report rather than from the page describing it, which makes years of accumulated research answerable, not merely downloadable. Reports can be found by their attributes, not only by the words in a title, and read in place, which is the difference between a document being published and being usable by the people it was written for.

Built with

• Drupal
• Custom RAG pipeline
• Vector similarity search and embeddings
• OCR text and layout extraction
• Hybrid vector and full-text search with Reciprocal Rank Fusion
• Result reranking
• Custom daily report import from external sources
• Report following and subscriptions
• Custom email notifications
• Drupal Search API extension
• Granular report search and filtering
• In-browser PDF rendering in JavaScript
• Multi-source content aggregation