That doesn't seem safe. What if there is a power outage and someone gets stuck during a fire or other catastrophe?
Also, to chime in on my experience, I've never encountered a handicap door that opens as easily as a regular one. They all have some sort of resistance, so I highly doubt it's a matter of improper installation.
I will admit to often using those handicap buttons when entering crowded buildings like universities, malls, etc., but I do it because it keeps the doors open for a minute or so, which means I don't have to hold them for people coming behind me.