One approach, which we’ve done in the past (also here), is that words indeed can cause harm. We may not understand how it works, we may no longer be able to use such power, but Hashem built into the teva such a phenomenon. This seems to fit the simple reading of Chazal that Bilam knew the moment of Hashem's anger and could summon a curse in that precise moment (Hashem, however, did a chessed and did not show any answer while Bilam was trying to do his thing.) Think of it this way: if you were to show someone in the ancient world a gun, they would have no idea how it worked either and would think of it as a magical device (brings back memories of A Connecticut Yankee in King Arthur’s Court). We can think of the idea of using words to cause harm in the same way (Ohr haChaim, Ralbag at the end of to'eles 6).
I think it may be a bit hard to convince modern readers that words can kill. Some other reasons Hashem stopped Bilam from cursing Bnei Yisrael:
1) Hashem wanted to stop Bilam so that Bilam would not be guilty of wrongdoing (Seforno 22:22. This begs the question of why Hashem should care so much about Bilam’s fate – let him sin and get the punishment he deserves?)
2) Had Bilam cursed Bnei Yisrael, every time Bnei Yisrael did wrong and were punished, the enemies of Bnei Yisrael would give credence to Bilam’s curses rather than attribute the punishment to Bnei Yisrael’s wrongdoing (Ralbag mentions this idea and rejects it). This is akin to Moshe’s argument in response to the cheit ha’eigel of of “lamah yomru Mitrayim…,” that the nations will attribute Bnei Yisrael’s downfall to Hashem’s lack of power c”v, not to their wrongdoing.
3) Had Bilam cursed Bnei Yisrael, it would make it very difficult for Bnei Yisrael to do teshuvah. Whenever they sinned and were punished by Hashem, rather than attribute punishment to their misbehavior, they would attribute it to Bilam’s curses. (Ralbag accepts this theory over the previous one, as it better fits Malachi’s description of Hashem's stopping Bilam being a chessed for Bnei Yisrael.)
4) Bilam’s curses would embolden the enemies of Bnei Yisrael to attack because they would perceive or assume that his curses have some effect even if in truth they cause no harm (Meshech Chochma).