What does the idiom lost her cool mean?

It means she "lost her temper".

Take this example: A boy goes to a girl and starts talking to her. She talks to him the first day. By the next day he begins to bug her or pester her so much that she doesn't want to talk to him or listen to him either. And by the third day she cannot stand him anymore and she yells at him.

In such a situation we would say "She lost her cool by the third day."