The size of the model depends on the number of parameters required to define it. Due to their nature, ensembles in general, and forests in specific require lot of parameters.
There are two workarounds,
- You can use other models that are defined using much smaller number of parameters, say SVMs.
- If you want to use ensembles and forests, then you can reduce the number of trees used, and you can reduce the number of leaves in a tree. This will however come at the cost of accuracy.
You should continue to use compact models irrespective of above workarounds. I would not recommend trying to implement the code for the model, as that is unlikely to give any significant improvements over the model.