There was a case in a in a session of my Homebrewy D&D 5e game that I played recently where I was playing a Gloom Stalker Ranger lvl 4 and was hidden from an enemy that was unaware of an incoming attack. In the spirit of the ambusher that is the Gloom Stalker my DM allowed me to role the extra d8, normally granted to me by Dread Ambusher feature on the bonus attack that I would get on the first turn of initiative order, for the damage that I dealt to the creature; similarly to a rogues sneak attack.

Should a Gloom Stalker Ranger always get a additional d8 when attacking an unaware creature outside of initiative order?

Mechanically speaking, it is not possible to attack an unaware creature outside of initiative order.

Such a situation should be resolved by an initiative roll in which the unaware creature is Surprised until the end of their first turn - and then the Dread Ambusher feature applies as normal, during the Gloom Stalker's first turn bonus attack.

Thematically speaking, if your DM adjucates surprise with an attack "outside of initiative", applying the extra d8 to a Gloom Stalker's attack is appropriate.

Yes

XGE p 42 says "at the start of your first round in combat" to describe the Dread Ambusher ability.

Making Attack actions without (or before) initiative having been rolled is a house rule, since PHB p 189 says that Initiative is rolled at the start of combat to determine the order of turns, and PHB p 192 says that you take the attack action on your turn.

My group uses a similar house rule (i.e. making attack actions without rolling initiative), we have a Gloom Stalker ranger in our party and that's how we've been running it. The "spirit" of the rule that we employ to adjudicate the situation is that they get to do one dread ambush, and that's at the start of the combat.

(However, if they were unable to take actions during their first turn, for example if they are Surprised, then they miss their chance to dread ambush.)

Yes.

Dread Ambusher (XGtE 42):

At the start of your first turn of each combat

The ambush necessarily starts combat and thus triggers the Dread Ambusher every time with that creature that is unaware of you and not part of any combat with you. The ambush is the first turn of the combat.