I agree 100% and have advocated for this since they were first released in 2007-2008(?)
The difference between dataset objects and other Toolbox specific functionality (IMO) is that the dataset object is/could be a fundamental building block that is application neutral - it is not a specific mathematical method that is only useful to a small user base. Anyone who has the need to store/process/manipulate/import/export tabular data of mixed types could benefit.
Im also someone who would not own MATLAB without the Stats Toolbox due to my work requirements, however I dont think that is reason enough to make the move to core MATLAB unnecessary. Core MATLAB functionality rarely includes features that are compatible with Toolbox specific objects. Toolboxes need to respect/handle core MATLAB features, but not the other way around. There are many functions in core MATLAB that could benefit from being able to handle dataset inputs/outputs.
I also would like to see the objects moved to core MATLAB so that they can be further developed/optimized. Access to dataset objects through either "." or "(.,.)" indexing is quite slow due to the object's subsref method being written in MATLAB. If this could be somehow rewritten the way it is for doubles, cell arrays, etc. I suspect performance could be improved considerably.