Three attempts: how Loggd merges movie and game data
How Loggd merges movie, TV and game data from TMDB, TVDB and IGDB into one dataset, told in three attempts, including the mistakes of the first two and quality numbers for 6.7 million entries.

Letterboxd has it easy. The app focuses only on movies and gets its data from The Movie Database, TMDB for short (FAQ). We wanted movies, TV shows and games in one app, and eventually books and music too. That's where the trouble started.
Why put ourselves through this? We believe taste doesn't stop at the edge of a medium. If you love horror movies, you often reach for horror games too. If you're into surprising plot twists in movies, you'll probably look for them in games or books as well. But Loggd can only use that if all media live in one shared dataset. When you open a movie in Loggd, you see the studio behind it and the people who worked on it, but also the games that go with it. This connection across media is what sets Loggd apart from apps that only know one medium. We cover the other features in a separate post about what makes Loggd different.
We built the dataset three times before we were happy with the result. Internally, today's version is called Nexus. The first two attempts weren't wrong, but neither of them convinced us in the long run. Whatever bothered us each time became a requirement for the next attempt.
Why one source isn't enough
There's no single database that covers every media type perfectly. TMDB is strong on movies and TV shows, TVDB on TV shows, IGDB on games and AniDB on anime. Each of these sources knows things you won't find anywhere else.
So you hook up several sources and everything's solved? Sadly, no. Each source delivers its data differently, each keeps its own entry for the same person or the same studio, and none of them knows that the others have "Star Wars" too. Loading the data is the easy part. Merging it is where it gets hard.
The first attempt was a copy of TMDB
We started small: only TMDB, only movies and TV shows. Games came later. We imported the data once and built the app directly on top of it, with search, images and detail pages. That was enough for the start, so you could log anything at all.
But it was a copy that ran in one direction, exactly once. There were no automatic updates, so the data went stale fast. There was no admin portal either. Whatever was wrong on TMDB was just as wrong on our side, and we couldn't do anything about it. A second source wasn't even considered in this setup.
The second attempt needed too much manual work
For the second attempt we reorganized the whole flow, and looking back, that was the most important step. Instead of one system, there were now two services: Ingest, which loads the data, and an admin portal where we process it.
Ingest was our intake. Picture a mail room: whether a supplier unloads a whole truck or drops off a few parcels, everything gets accepted and sorted here, and someone notes what's new. The suppliers could hardly be more different. IGDB delivers around 50 tables as CSV exports, TMDB sends individual JSON objects, AniDB offers a UDP interface. Behind Ingest it all looked the same, in a single, strict schema. Basically, Ingest was a smart cache that returned current results for whatever we asked for. We could ask as often as we liked without flooding the providers with requests.
In the admin portal we defined what a movie, a TV show or a game looks like for us, each as a JSON schema. Transformers brought the data from Ingest into that shape. Then came the part that would later slow us down: we checked every change ourselves and copied it into our database by hand. From there everything ran automatically, into the search index, into the backend for logging and to the hosted images.
Back then we separated loading from processing, and we still stick to that today. The rest slowed us down over time:
The system was technically very slow and error-prone.
With this much data, merging every change by hand was barely doable.
We self-hosted a lot of dead data, meaning low-quality entries along with their images. We spent money on that without it helping anyone.
The third attempt is called Loggd Nexus
For the third attempt we kept the skeleton: collect first, then process. We rebuilt everything that was tied to the three weaknesses.
The slowness we fixed with a different language: the engine is now written in Go. The transformers for the individual sources change often, so they are small TypeScript functions. As regular code, they can be versioned and reviewed, and AI tools get the context they need when writing them.
The intake got stricter too. A row only counts as changed if its content really changed. Fields that fluctuate all the time, such as ratings or vote counts, stay out of the checksum. Otherwise every new vote on TMDB would trigger the whole processing chain behind it again.
A sync also either delivers complete data or fails. If a new dump is missing more than 10% of the previous rows, we delete nothing. In that case the download is probably broken, and the catalog hasn't actually shrunk.
After that, every changed row goes through five steps: transform, filter, match, merge, enrich. In the second attempt, matching and merging cost us the most manual work, so we'll start there.
Does this entry already exist?
Merging sounds easier than it is. A movie comes with a list of the production companies and actors involved, and so does a TV show or a game. If we just glue these lists together, companies and people show up twice, because every source keeps its own entries. If we're too generous instead, two different movies with similar titles get merged into one.
Both are bad, but not equally bad. A duplicate can be merged later. A false merge, on the other hand, has mixed up data from two works, and pulling that apart cleanly is far more work. This trade-off is behind everything that follows.
Imagine every movie in Loggd has an ID card with our own number on it. Every source that knows the movie writes its own number on the card and leaves a receipt like "my row X has IMDb ID A and Wikidata ID B". These receipts tell us which external IDs belong to which Loggd entry. Internally we call this the ID graph.
When a new record comes in, nobody sorts it in by hand anymore. Every candidate goes through the same stages until one of them makes a confident decision:
Shared IDs. If an IMDb, TMDB or Wikidata ID already belongs to exactly one entry, the case is clear.
Title search. Without a shared ID, we run a fuzzy search on the title, with a tolerance of one year. A hit that has a different strong ID than the candidate is thrown out right away. This is one of our most important safeguards against false merges.
Decision model. For the remaining candidates, a model estimates how likely it is that they're the same entry and how confident it is about that. At 80% probability or more with at least 60% confidence, we merge. If it's at least 85% sure there's no match, we create a new entry.
Review. We look at everything in between ourselves.
With people we're even more careful and only merge them through shared IDs. Among millions of people, simply too many share the same name for a name alone to prove anything. So here we knowingly accept duplicates to avoid false merges.
Across media, you see this best with collections: franchises like Star Wars that include both movies and games. Here the model also answers whether one collection is the parent of the other. That's how TMDB's "Star Wars Collection" and IGDB's "Star Wars" games become one shared collection. And if something does get linked wrongly, every automatic link can be undone.
Who wins when sources disagree
Now we know which movie this is. That leaves the question of what we take from the new delivery. For that we use a three-way merge, the kind you know from Git. We compare three states: what this source delivered last time, what's currently in the entry and what the source is delivering now. If only one side changed something, we take that change. If two sources truly contradict each other, the model decides, and if it's unsure, we decide.
There's one exception. If a value is about to disappear from a movie or TV show, that always goes to review. Otherwise a source that loses data could silently delete fields on our side.
What we've corrected is locked. Every manual change and every review decision locks the affected field. The pipeline never overwrites a locked field, no matter what a source delivers later. In the first attempt we couldn't correct anything at all. Without this rule we could, but every correction would be gone with the next import, and all that manual work would be wasted.
How much we leave to AI
More sources make the data more complete, but also more crowded. Our goal is a dataset that is as complete as possible and still stays clean and lean, free of duplicates, adult content and junk data. A model helps us with that, but we don't trust it blindly.
That's why our decision model only answers clearly defined questions, such as "Is this the same movie?", and returns a probability and a confidence with each answer. It checks movies and TV shows for adult content the same way. A title with a score up to 0.15 passes, from 0.65 on it's blocked, and in between we take a look ourselves. If a source gives a clear signal, such as a standard age rating, we don't ask the model at all.
Where do these thresholds come from? We measured them. For matching, the ID graph helps again: where two entries provably share the same strong ID, we know they belong together. That shows us how often the title search and the model get it right. New sources also start out in shadow mode. The pipeline runs all the way through but writes nothing except the matching decisions, and we calibrate the thresholds on those before the source goes live. That's where AniDB is right now.
A second model translates descriptions into English and German and suggests tags. That taught us how quickly a model overdoes it. It handed out content warnings and plot features so generously that we removed both again. Since then it's only allowed to fill empty fields and never overwrite a value from a source.
Dead data stays out
That leaves the last weakness from the second attempt: all the dead data we paid to host. Today we re-score every entry after every change. Each type has a list of weighted rules that produces a quality score between 0 and 1. A movie also needs a name, an English description, at least one poster and a backdrop. If any of these is missing or the score is below 0.2, the movie stays disabled, and you won't see it in the app.
People and companies only show up if a visible movie, TV show or game references them. That way, millions of people who only appear in some source automatically stay out. We now host images only for visible entries. And a few seconds after a change is done, it's in the search index the app reads from.
How good the data is in October 2026
In October 2026, Loggd Nexus holds about 6.73 million entries. We sort them into three tiers by quality score: "good", "adequate" and "sparse". For games, an entry below 0.5 counts as sparse and one at 0.75 or above as good. For all other types, the cutoffs are 0.2 and 0.5.
| Type | Entries | Disabled | Good | Adequate | Sparse |
|---|---|---|---|---|---|
| People | 4,602,003 | 22.6% | 5.3% | 8.6% | 86.1% |
| Movies | 1,191,430 | 57.0% | 12.5% | 73.2% | 14.3% |
| Games | 341,822 | 19.7% | 18.1% | 69.2% | 12.7% |
| Companies | 340,369 | 29.0% | 3.9% | 11.6% | 84.5% |
| TV shows | 232,115 | 69.3% | 10.0% | 56.7% | 33.3% |
| Collections | 21,062 | 93.8% | 24.4% | 6.4% | 69.1% |
| Engines | 1,505 | 0.0% | 0.5% | 28.7% | 70.8% |
| Platforms | 220 | 0.0% | 3.2% | 80.0% | 16.8% |
| Total | 6,730,531 | 30.7% | 7.4% | 24.9% | 67.7% |
The quality shares refer to all entries of a type, including disabled ones. The total also includes five events.
Movies and games are in decent shape: more than 85% of their entries are at least adequate. Still, 57% of movies and 69% of TV shows are disabled, because a missing English description or a missing backdrop is enough. If only the image is missing, the entry becomes visible as soon as one arrives.
People and companies look very different: 86.1% and 84.5% of them are sparse. Almost 94% of collections are disabled, because a collection only appears once at least one of its movies or games is live. Most of our remaining work is here. We mainly want to close these gaps with more sources such as Wikidata.
Why we build all this
With a single source, like Letterboxd, a lot of this would have been easier. But then movies, TV shows and games would have stayed separate worlds. Only the shared dataset links movies and TV shows through the people they share, and all three media through studios and collections. That's what the cross-media recommendations in Loggd build on, and so does the group chat activity that suggests a movie or game to everyone in the group.
Ideally, you won't notice any of this in the app. The data should simply feel complete and clean. You can see what that looks like in the current version at loggd.me, which we moved from React Native to Flutter this year. And if you spot anything that looks off, let us know on Discord. Whatever we fix after that is locked, and no source will break it again.