I have coffee sometimes.
Various zoners will answer this question differently. I am of the mindset that if it isn't excessive, it is OK.
You do have to pay attention to how you respond to that cup of coffee, i.e. does the caffeine cause you to experience hunger any earlier than if you had not had the coffee? If so, then it is better to wean off of the coffee. However, if the coffee does not cause early hunger, then you may be fine.
Also, as far as the milk goes, if you are only splashing enough in there to lighten your coffee, then you probably don't have to count it. But, if you are using 1/2 cup or more, then count it in to your block counts.