You can do whatever you want with your life. If you’re not comfortable with going, then you shouldn’t go.
You should also ask yourself if you’ll be ok in the future with the idea that your and your brother’s opinion about a third person, who isn’t a part about either of your lives, is a valid ground for potentially breaking forever your relationship with your brother. Maybe simply avoiding this topic that creates tensions would be enough?
You can do whatever you want with your life. If you’re not comfortable with going, then you shouldn’t go.
You should also ask yourself if you’ll be ok in the future with the idea that your and your brother’s opinion about a third person, who isn’t a part about either of your lives, is a valid ground for potentially breaking forever your relationship with your brother. Maybe simply avoiding this topic that creates tensions would be enough?
Neither of you is an asshole.