Python linear least squares function not working

Question

Ok, so I'm writing a function for linear least squares in python and it's pretty much just one equation. Yet for some reason, I'm getting a ValueError. My best guess is it has something to do with the .reshape function, since in this question I had a very similar problem and reshaping was the solution. I've read up on it and from what I gather, w in my function is in format (n,) and the result would be in (n,1) as in my previously mentioned question. I tried reshaping x_train and y_train but I only got an error that I can't change the size of the array. I guess my parameters were set wrong. Right now I'm lost, and I have many more functions like these to go through -- I wish I could understand what am I missing in my code. The equation seems to be in order, so I suppose there's something I should be adding everytime - possibly the reshape function cause I'm still using the same models as in the last situation. I hope it's the right place to post this question, I don't know what else to do but I really want to understand so I won't have these problems in the future, thank you.

Code (np. stands for numpy):

def least_squares(x_train, y_train, M):
    '''
    :param x_train: training input vector  Nx1
    :param y_train: training output vector Nx1
    :param M: polynomial degree
    :return: tuple (w,err), where w are model parameters and err mean squared error of fitted polynomial
    '''
    w = np.linalg.inv(design_matrix(x_train, M). * design_matrix(x_train, M)) * design_matrix(x_train, M).T * y_train
    err = mean_squared_error(x_train, y_train, w)
    return (w, err)

design_matrix and mean_squared_error are working just fine. Traceback:

ERROR: test_least_squares_err (test.TestLeastSquares)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "\content.py", line 48, in least_squares
    w = np.linalg.inv(design_matrix(x_train, M).T * design_matrix(x_train, M)) * design_matrix(x_train, M).T * y_train
ValueError: operands could not be broadcast together with shapes (7,20) (20,7)

Answer 1

Assuming that design_matrix returns a matrix, this code

design_matrix(x_train, M).T * design_matrix(x_train, M)

most likely does not do what is intended since * is performing element-wise multiplication (Hadamard product of two matrices). Because your matrices are not square, it thus complains about incompatible shape.

To obtain matrix-matrix product, one might do (assuming numpy was imported as import numpy as np):

np.dot(design_matrix(x_train, M).T, design_matrix(x_train, M))

Similar reasoning then applies to the rest of the statement * design_matrix(x_train, M).T * y_train...

Also, you might want to evaluate design_matrix only once, e.g., to put something like

mat = design_matrix(x_train, M)

before the line calculating w and then use merely mat.