An agent-based simulation of generation company behavior in electricity markets is developed. Learning dynamics of companies is modeled with an extended Q-learning algorithm. Different market clearing mechanisms of the regulator are compared. Convergence to Nash equilibria is analyzed under different cases. The level of competition in the market is studied.