Spouses - how do you talk about drugs with your partner?

I've been wondering about this ever since I went back to my SO and I have been sober for a while now. I did have some weed as I am able to manage my use of it, but I have been off the hard drugs for a while. He's able to admit that he has used, and I know he still does, but he doesn't want me to know about it. I have some hard feelings here, as he did have some concerns about my drug use as well. Should I confront him about it, or am I able to let go of the need to know?