I was reading an article earlier today, and it brought me back to a question I’ve heard over and over again in real data/infra teams:
Do we just accept vendor lock-in because it’s convenient,
or do we take the pain and build an open, multi-engine metadata stack?
For context (not my product, just what triggered the thought):
https://medium.com/p/35cc5b15b24e
I’m not trying to argue Gravitino vs. UC here — I’m more interested in the architectural mindset behind these two approaches.
On the vendor-integrated side, the upsides are obvious:
smoother UX
one place for lineage/policies
fewer moving parts
But so are the downsides:
cost keeps creeping up
you end up tied to one engine/format
migrations basically don’t happen in real life
And on the open/composable side:
Spark/Trino/Flink/Ray all first-class
Iceberg/Hudi/Delta can actually coexist
Metadata isn’t tied to compute
But again:
inconsistent metadata models everywhere
no unified governance layer
someone eventually owns a pile of glue code forever
So I’m curious: what actually works in practice?
If your company had to make this choice:
Did you go all-in on a vendor, or build something open?
Did the decision age well after a year or two?
Has anyone actually avoided metadata sprawl without getting locked in?
Where do lineage, ACLs, policies, and the “source of truth” actually live in your setup?
Really interested in what folks think, especially if you're juggling multiple engines, table formats, and clouds.
My take from working on free and opensource Godot Engine and 3d formats metadata is that the main difference is if the people have the knowledge / knowledge transferred of how the process works.
If you lost the knowledge and are substituting a library (vendor) for that knowledge, you have to rewrite that library to understand its gaps and how to update it.
For example let's say you have a free and opensource 3d formats pipeline.
For a digital content creation tool studio (dcc) which has one tool (maya) and they use a intermediate format called (fbx) and you want to interchange a 3d avatar in Godot Engine.
If you want to amend the process to swap out maya with blender, you would need to understand how the fbx format works and also how maya, blender and Godot Engine works.
Sure you can outsource the library to an external project like assimp, but the moment a particular fbx is broken you basically start rewriting assimp. If the errors are close to 80% of the imported cases, you'd need to rewrite assimp.
Also fbx in Godot Engine is a reimplementation of FBX as there's no specifcation of FBX file format. This is similar to your vendor locked in description.
This isn't a typical enterprise data exchange process but maybe my change of the theme of the process can help.
I was reading an article earlier today, and it brought me back to a question I’ve heard over and over again in real data/infra teams: Do we just accept vendor lock-in because it’s convenient, or do we take the pain and build an open, multi-engine metadata stack? For context (not my product, just what triggered the thought): https://medium.com/p/35cc5b15b24e I’m not trying to argue Gravitino vs. UC here — I’m more interested in the architectural mindset behind these two approaches. On the vendor-integrated side, the upsides are obvious: smoother UX one place for lineage/policies fewer moving parts But so are the downsides: cost keeps creeping up you end up tied to one engine/format migrations basically don’t happen in real life And on the open/composable side: Spark/Trino/Flink/Ray all first-class Iceberg/Hudi/Delta can actually coexist Metadata isn’t tied to compute But again: inconsistent metadata models everywhere no unified governance layer someone eventually owns a pile of glue code forever So I’m curious: what actually works in practice? If your company had to make this choice: Did you go all-in on a vendor, or build something open? Did the decision age well after a year or two? Has anyone actually avoided metadata sprawl without getting locked in? Where do lineage, ACLs, policies, and the “source of truth” actually live in your setup? Really interested in what folks think, especially if you're juggling multiple engines, table formats, and clouds.
My take from working on free and opensource Godot Engine and 3d formats metadata is that the main difference is if the people have the knowledge / knowledge transferred of how the process works.
If you lost the knowledge and are substituting a library (vendor) for that knowledge, you have to rewrite that library to understand its gaps and how to update it.
For example let's say you have a free and opensource 3d formats pipeline.
For a digital content creation tool studio (dcc) which has one tool (maya) and they use a intermediate format called (fbx) and you want to interchange a 3d avatar in Godot Engine.
If you want to amend the process to swap out maya with blender, you would need to understand how the fbx format works and also how maya, blender and Godot Engine works.
Sure you can outsource the library to an external project like assimp, but the moment a particular fbx is broken you basically start rewriting assimp. If the errors are close to 80% of the imported cases, you'd need to rewrite assimp.
Also fbx in Godot Engine is a reimplementation of FBX as there's no specifcation of FBX file format. This is similar to your vendor locked in description.
This isn't a typical enterprise data exchange process but maybe my change of the theme of the process can help.
> This isn't a typical enterprise data exchange process but maybe my change of the theme of the process can help.
A typical enterprise data exchange would be like parts and suppliers for inventory.