You haven't defined a variable named hamming in your code, but there IS a function named hamming in Signal Processing Toolbox. Therefore MATLAB interpreted this line of code:
wy = y'*hamming:N-1;
as calling that function with 0 input arguments and using that in the calculation of wy. But the HAMMING function from Signal Processing Toolbox requires at least one input, the number of points in the window.
Either define the variable named hamming before you try to use it or call HAMMING with one or two input arguments (the documentation describes the meaning of each input argument.)
