Every good SF or Fantasy story needs its monsters or aliens, but it is not enough to simply slap together something ugly and then drop it in front of the characters. To make your story believable in any setting, the monsters must feel as real as everything else.

First off, the creature has to make sense. Make it creative, inventive, and unusual looking, but it still has to be something that either could have evolved given the local conditions, or the conditions of wherever it first evolved from. In short, it needs a biology, or at least a basic idea of one. A classic example would be the tremendously huge dragon, with the question of “what does something that big eat?” Now in a fantasy setting, the answer could very well be “it feeds off of the local magic fields,” or “this type eats rock and excretes crystals.” As long as there is something within the context of the story setting to explain it.

Next you need a reason for it being there. If it’s a natural denizen then it must have a place in the local ecology. Or is it set there as a guardian? If it isn’t native then where did it wander in from and why? Any creature needs a good reason for being where they find it, something that makes some sort of sense in the given context.

Then comes motivation. Not all monsters will automatically attack anyone that comes along. Is it there to hunt? Or maybe it was sleeping and the characters woke it up. Some critters may even be friendly if the characters in the story approach it right, work as an ally of sorts. And use a bit of logic here. Even a hungry mountain lion is not going to attack a strong well armed group of people; more likely it will pick off someone straying out of sight of the main group. Giving even a random beastie a suitable motivation will add that much more depth to a story.

Of course, in most cases you needn’t spend that much time puzzling all the ins and outs of a simple encounter for the story’s characters. Just a couple of quick notes and a little thought, but the difference will mean fewer readers crying, “Ah, come on. That’s just ridiculous!”