What is penalty?

It is a kind of damping factor applied to your rankings.
How does one know if his site is penalized? No way.
Your site may be penalized right at this time and you don't know it.
You should understand though that not every loss of rankings is a penalty!
However, Google does penalize sites that use blackhat SEO techniques such as hidden links, keyword stuffing or having bad link neighbourhood or irrelevant links.
There is no artificial penalties made for the only sake of dropping some sites down.
Google is about search, and it worries about search, not about putting some site higher or lower in rankings.
So if your site has received a penalty you should ask yourself, why Google thinks that my site is not relevant to that query?
Is this something about my content?
Is this those links I bought last month?
There is always a reason.