The first rule in life in Japan as a foreigner is to always, ALWAYS carry your RESIDENCE CARD (在留カード, or zairyu kaado, or “gaijin card” colloquially).
This is the single most important document for your daily life in Japan. It will be your primary form of I.D. and may be requested of you at any time by police officers. Failure to keep it on hand can unnecessarily complicate matters.
Make sure you carry your zairyu card (aka “gaijin card”) at all times. I know it can be annoying if you want to go out for a bike ride or just down the street to buy something from the vending machine, but police are legally allowed to stop you and ask for it at any time. For example, I have been stopped in an airport waiting for my dad to arrive for a visit, and multiple Tochigi JETs were stopped soon after the 2015 Paris bombing. (Yes it encourages racial profiling, but it’s best not to make a scene and just cooperate.)
However, if you feel that something is not right with the situation and suspect that the officer is a fake, make sure you know your rights. Please everyone read this article–it has a lot of details regarding what to do if you suspect a fake or unlawful police officer.