The olive oil would in place in butter or butter substitute. Some people don't even use milk or a milk substitute in mashed potatoes, but simply use some of the cooking water for the liquid to make them smooth. You can add parsley, chopped spring green onion, and other seasonings to flavor those mashed potatoes.
They can be really yummy and very nutritious. Choose from things in your spice cabinet that you like the flavor(s) of and experiment. Potatoes are cheap!... not much money to lose with experimenting and lots to gain from having tried expanding your potato horizons.