You've just gotta own that you are being judgmental, not about his worth as a human being, but as a potential partner for you. Hopefully he'll call me back and we can have a quick chat. You say you "think you can deal with" once a year pot smoking, this choice of phrasing makes me doubt that. You directly and calmly stated your thoughts on a certain manner, rather than keeping it hidden away, and it worked out better for everyone involved. But it's not a sin to go on a date because you aren't sure and you might have fun, to say nothing of changing your mind.
I'm glad I asked because he thought I was on board with it too and was surprised to hear that it was a deal-breaker for me.
The #1 Dating Deal Breaker of 2016
Predicting his reaction is not possible from a two paragraph mefi exposure. If he's comfortable enough to bring up smoking in Amsterdam on the first date, he definitely has a different outlook on pot than you do. But if I were in Amsterdam, then I might smoke it there just to check that box on the tourist to-do list. Say what you just said here. It's not weird and it's not judgmental.