Why -O2?
Why not -O3, -O4, -Ofast, or -ffast-math? -Os is almost the same as -O2, but focuses on size reduction more, which is a concern you mentioned; why not use it?
Or even better: Why not let the end user decide if they want/need the additional flags, and let them edit the makeline themselves, if they choose to do so? One edit after install, and you can have the custom flag set which suits your needs from that point on.