The "Regression Learner" app does not currently support specifying custom Kernel functions. However, if you are willing to consider a command-line approach, you could use the fitrsvm function to fit a Regression Support Vector Machine. In the inputs for fitrsvm, you can specify the "KernelFunction" Name-Value pair to point to your custom Kernel function. Please refer the following link for more information:
As an alternate workaround, you can generate code for your regression model from the Regression Learner App using the "Export Model" drop down. You can modify the "KernelFunction" property in the generated code to point to your custom Kernel function. You might have to also modify other Kernel associated parameters such as "KernelScale" accordingly as required by your use case.
On a different note, I work for the MathWorks and I have forwarded this use case to the appropriate product team.