Mfcttrf

I am writing a fiction short story roughly based off medieval times in Europe. In it I have a group of men who live in a castle and fight for the castle's lord. (They mainly protect the surrounding villages from a group of bandits.)
If they existed in modern day times I'd refer to them as soldiers. They are much like traditional knights, only they're a whole group, they've never been knighted and aren't members of nobility.
What might they be called? I'm looking for a term, from sometime in the medieval time period (I don't care when), that the surrounding villagers would call them by.
Are the terms vavasseur or sergeant appropriate? Is there another better term?

They are Men at arms, professional soldiers.

The lance fourni of a knight, the unit of soldiers a knight brought to battles or protected the fief, was formed by professional horse-riders, archers and/or spearmen. They were also called men at arms, so the sergeant/marischal in charge of your unit could be a knight or a common man (a yeoman if you heavily borrow from English history).

As Carlos Martin has noted, these soldiers are men at arms. They might be armed with swords, bows or crossbows, spears, or halberds depending on precise period and geographic origin. Generally they would be responsible for their own arms and armour, but a wealthier lord might choose to upgrade that in order to field a more impressive and effective force.

Groups numbering in the range 10-20 would be commanded by a sergeant, and might be termed a peloton (the French root of our modern platoon), troupe, or escadre (French root of modern squad) without being too anachronistic. Larger groups, composed of multiple pelotons, would be termed a company and commanded by a captain assisted by a lieutenant. Within a single levy, either mercenary or feudal, there would be some consistency in peloton size, but there need not be complete consistency between companies of different origin, other than the general pattern just described. In companies of several pelotons the most senior sergeant would be the sergeant major.

If the company is of such a size to fight as two separate wings, the captain and lieutenant would command one each. If large enough to deserve three wings, then the captain would command the centre with the lieutenant and sergeant-major each commanding a wing to his left and right. A mercenary company might be as large as two or three hundred men in the later part of the period.

This answer about living arrangements for a castle's Constable notes that the castles built in Wales by Edward I generally were designed to support a garrison of slightly more than 30 men.