It's strange how some countries "get it" and others don't.
The Irish get it; the Welsh too.
But England, and probably Scotland as well, the Patron Saint thing just hasn't gripped the nationalistic psyche to the same extent.
I am hugely Anglophile in a positive way as opposed to a protagonistic way, but St George's Day just hasn't grabbed me, nor many others.
I don't see any harm in celebrating it should you feel motivated so to do; but I don't think of myself as any less patriotic because I don't.