A modifier is a number that changes a die result or a natural roll, it can be positive or negative. Any modifier that is always positive or absent is called a bonus. Any modifier that is always negative or absent is called a penalty. It is also possible (and common) to say that a character "has a size penalty" instead of saying that (s)he "has a negative size modifier" (or "has a size bonus" instead of "has a positive size modifier").
Usually the higher the modifier, the better, and the bigger the penalty, the worse.