AI & context
7 min

How to Evaluate a Knowledge Graph Platform

Key takeaways

  • Shortlists in this market mix product categories, so the feature grid compares things built for different jobs.
  • Test four things: portability, governance depth, query-time resolution and expert usability.
  • Scale claims rarely separate anyone. Most governed models are far smaller than vendor benchmarks.
  • An evaluation with no named future owner optimises for the purchase instead of the operation.
In this article
Share

How do you evaluate a knowledge graph platform without wasting a quarter?

Decide what the graph is for before you look at a single product. Evaluations in this market go wrong early, and almost always the same way. A shortlist gets assembled from search results. The products turn out to be in different categories. The feature grid that follows compares things that were never trying to do the same job.

So the evaluation has two halves. Establish the category, then test the two or three things that are hard to change later. Everything else is detail that a trial will settle on its own.

Which category are you actually buying?

Six product categories share this vocabulary, and a product page rarely settles which one you are looking at.

If the problem isThe category is
Disagreement about what terms meanOntology and taxonomy management
Querying connected data at volumeA graph database, RDF or property depending on portability needs
Finding and tracking existing assetsData catalogue
Consistent metrics across BIMetrics semantic layer

Most organizations eventually need more than one, and the failure is buying from one category expecting coverage of another. Knowledge graph vs data catalog covers the most common of those collisions, and knowledge graph vs graph database covers the other.

What should the trial actually test?

Four things, in descending order of how expensive they are to get wrong.

The four tests

  • Portability. Build a small governed model, export it, diff the export, then re-import into a different product. The fourth step is the one everyone skips and the only one that proves anything.
  • Governance depth. Change a definition, then ask the product what it said before. Many have versioning, an audit log and approvals as three separate features with nothing joining them.
  • Query-time resolution. Call a term from outside the product, timed. If the answer involves an export or a nightly job, no retrieval system will use it.
  • Expert usability. Can a domain expert change a constraint without filing a ticket? If not, every definitional change queues behind engineering.

Notice what is absent: scale. Vendor benchmarks run to billions of statements and most governed models are orders of magnitude smaller. Almost every product in the category will handle your volume, so scale claims rarely separate anybody.

What does a demo not tell you?

Demos run on data chosen because it works, and the three things that decide the next five years are all invisible in one.

Whether your model fits the product's assumptions. Many tools assume a single parent per concept, and your domain will eventually disagree. Load your own vocabulary and find out in week one.

What the export actually contains. Every product claims export. The question is whether constraints, alternative labels and provenance survive it, and the only way to know is to diff.

How fast approval really moves. A workflow that looks clean in a demo can take a fortnight in practice once real approvers are involved. Latency is a governance property and it is the thing that determines whether people route around the system.

Who needs to be in the evaluation?

More people than usually are, and the missing ones are predictable.

Who has to be in the room

  • A domain expert who will actually approve definitions. Not a proxy. They will tell you within ten minutes whether the editing experience is tolerable.
  • Whoever runs the consuming system. They care about resolution latency and API shape, and nobody else will raise it.
  • Someone who has migrated a platform before. They will ask the export question that the rest of the room finds tedious.
  • The person who will own this in year three. Often nobody has been named, and that is itself a finding.

That last gap is worth taking seriously. An evaluation with no named future owner tends to optimise for the purchase instead of the operation. The product that wins is the one that demos best.

How long should this take?

Four to six weeks from shortlist to decision, and longer than that is a signal rather than diligence.

A workable shape

  • Week one: category. Establish which of the six categories each candidate is in, and cut anything in the wrong one. This usually halves the list.
  • Weeks two and three: trial on your own model. Two products, not five. Run the four tests.
  • Week four: the export and re-import. Give it its own week, because it is the step that gets squeezed out and the one that matters in year four.
  • Weeks five and six: decide and write down why. With trial evidence against each weighted criterion.

Evaluations that run six months are rarely more rigorous. They are usually stalled on a decision nobody is empowered to make, and extending the trial is how that gets disguised as process. If the timebox slips twice, the thing to fix is the decision right and not the testing.

How should you score it?

Weight the four tests heavily and almost everything else at zero.

Feature grids in this category run to a hundred rows, and ninety of them are satisfied by every serious product. Scoring them equally produces a total that looks rigorous and moves with whoever wrote the grid. A short weighted list is both more honest and harder to argue with.

Record why each score was given, with evidence from the trial instead of from the vendor's documentation. When somebody asks in two years why this product was chosen, the answer should be reconstructable. Questions to ask before buying covers the checklist version, and knowledge graph ROI covers measuring whether it worked, and knowledge graph tools maps the categories your shortlist is drawn from.

So what should I take away?

Establish the category first, then test portability, governance depth, query-time resolution and expert usability on your own model. Ignore scale claims. Ignore feature counts.

TopQuadrant sits in the ontology and taxonomy management category, so we are one option and not the whole shortlist. The honest version of our position is narrow: we built for the agreement and the record of who made it, on open standards, so the model can leave. Test that the same way you test everyone else. Export, diff, re-import. Our knowledge graph platform should survive that test, and if it does not we would rather you found out in a trial.

Give your AI the context it's been missing

See how the TQ Data Foundation turns your enterprise knowledge into trusted, Al-ready context.