My husband told me that drinking a cup of coffee is like having sex outside of mairage. I told him he was full of it.
Upon first reading this question, I do believe that you had Gramps completely at a loss for words.
So of course my first knee-jerk reaction was to come up with a couple of snarky answers, such as:
You may tell your husband that Gramps says he’s full of it.
If you ever see your husband drinking coffee, you have my permission to file for divorce from that adulterer.
Now in all seriousness, not knowing in what direction your husband was taking this, I can respond only in this way. Both of those will keep you from going to the temple. We must follow the Word of Wisdom, and we must obey the Law of Chastity.
However, that being said, only one of those would result in a possible disciplinary action that could involve being disfellowshipped or excommunicated. That would be the breaking of the Law of Chastity.
So as you can see, there is no way to liken the two of them. Obviously breaking the Law of Chastity has far more serious consequences as far as membership in the Church goes.
I hope this helps.