How real AI Engineers build MCP Servers

For the curious builders tired of surface-level solutions.

A few months back, I got curious about MCP servers. So, I did what I always do:

Deep dive mode 🤿

Tutorials, GitHub repos, Substack articles, LinkedIn posts, … anything I could get my hands on.

But there was one tiny problem.

Everywhere I looked, it was the same story: everyone was just connecting to an MCP server someone else had already built … usually hooking it up to Claude Desktop.

Great for beginners. Not so great if you actually want to build a custom MCP Server 😅

So, Alex Razvant and I decided to change that when we kicked off the Kubrick project.

We built something for the builder community: a project that doesn’t just connect to an MCP server; it shows you how to build an advanced one from scratch with frameworks like Pixeltable, FastMCP and Opik.

Hands-on, no-fluff content for people who want to get their hands dirty and build for real.

This article builds on the theory and code covered in previous ones, so be sure to check them out if you haven’t already!

• Lesson 0: Your first Video Agent. Multimodality meets MCP

• Lesson 1: The future of Video Search is here

Ready? Let’s go! 👇

---

Our multimodal MCP Server

Video RAG System + MCP Server working hand-in-hand

Last week, Alex and I covered everything related to the multimodal pipelines for our Video RAG system.

In short, we’re using Pixeltable to build three separate embedding indexes, one for audio transcriptions, one for the video frames, and another for the captions of those frames.

The future of Video Search is here

But now that the indexes are set up, how’s Kubrick actually going to get to all that info? I mean, we need a way for our agent to interact with these indexes, right?

You guessed it. Through an MCP server! 😂

Implementing our MCP Server using FastMCP

As you can see above, we’re using the FastMCP library to build the MCP Server.

In particular, we’re exposing three main entities through it: resources, prompts, and tools.

Let’s break each one down.

If you’re curious about how the server’s implemented, you can check it out here!

---

MCP Resources

MCP Resources can be accessed directly through the MCP Inspector.

Resources represent data or files that an MCP client — like MCP Inspector, Claude Desktop, or the Kubrick Agent — can read.

In our case, we just needed to expose some info about the Pixeltable tables available in the system.

This way, Kubrick can keep track of which videos have already been processed and are ready to use.

---

MCP Prompts

Versioning the MCP prompts using Opik

If you look at most MCP projects out there, you’ll notice that pretty much all of them only define tools.

So, in most cases, the prompts end up being managed inside the Agent’s code.

But for us, we wanted to keep both the tools and the prompts together in the same MCP server. That way, any changes to the tool logic or the prompts happen in one place.

On top of that, we wanted to version our prompts too, and we did that using Opik. You can see how simple it is to version a prompt with this framework in the snippet below.

By simply defining the prompts as opik.Prompt classes, we’re able to version them easily.

---

MCP Tools

Exploring MCP Tools with MCP Inspector

And here they are.

Finally, let’s talk about the tools 😎

These are the real powerhouse behind Kubrick’s video processing abilities. We’ve defined four main tools.

Tool 1 - process_video

When you upload a new video in the Kubrick UI, this tool is called automatically.

It fires up the Pixeltable processing pipeline (the one we covered last Wednesday) — extracting audio, transcribing, pulling frames, captioning, and so on.

By the end, the video is fully searchable by Kubrick.

Tool 2 - get_video_clip_from_user_query

Just like the name says, this tool picks out a clip (5–10 seconds) from a longer video based on the user’s query. Here’s an example of it in action.

Tool 3: get_video_clip_from_image

Same idea as above, but instead of a text query, you pass in an image.

Tool 4: ask_question_about_video

Got a general question about the video? This tool handles that. It taps into the captions and transcriptions embedding indexes to find the answer for you.

---

And I think that’s enough for today!

I could go on about this project all day long, but hey — now it’s your turn to try it out. We’ve got a full Python project for this MCP server implementation, and I’d really encourage you to dive in and explore the code.

You know in The Neural Maze we strongly believe in learning by building, so go check it out! 💪

Also, keep an eye out this Saturday — Alex will be dropping another deep dive article in his Neural Bits publication, so stay tuned for that.

Let me know what you think about our MCP Server setup.

Got any suggestions for improvement? Any questions?

I’d love to hear your thoughts! 🙂

---