Axes are not polymorphic--they require a given data type.
You'll have to create two axes and select which to display depending upon which type of data is to be displayed.
If you need both simultaneously, you'll have to have two x axes on the same figure or use two figures.
The only (crude and not recommended) way around would be to use the venerable datenum instead of datetime class. datenum is just a double camoflauged as a date; to make the axis represent dates one has to use the deprecated and klunky datetick function. As noted, NOT recommending this, but it is a way to put either on the same axis.