I like to pretend to a certain amount of smirking deviousness - I once derailed an "evil campaign" by convincing the GM to put his BBEG in the party as a mole... then convincing the resident assassin to kill him as a warning to his "boss" (he didn't know he had just killed the boss, it was hilarious). Since you're approaching things as a GM, I'll tell you how I do devious, and what our GM did about my machinations.

1. Risk Assessment - when I plan for evil, I first figure out what I can get away with... and what I can afford to lose should I fail. If your Big Bad can't afford to be stopped at this point in the game, don't put him in a position to be stopped. If that doesn't work: Our GM asked for a minute to think, and pulled on his beard while staring at the headless corpse of his villain (I was thorough). Then he nodded and said, "Yeah, he's got nothing for that."




2. Plan For Failure - Make your plan foolproof all you want to, someone who isn't a fool will tear it apart. It's better to have a retreat planned should things go south than to bet big and die. If that doesn't work: Our GM decided his villain didn't have such a plan... but another villain did, and had them step into the BBEG role, having waited for "that simpering fool to foul things up, as I knew he would." I can't prove that the new villain wasn't just the old villain with the name hastily rewritten, but it was what I would have done in his place. It's a prime example of #3:




3. They Don't Have to Know It Wasn't Planned - a trademark of megalomania is taking credit for things you aren't responsible for. For example, I didn't actually think I was going to get the BBEG killed. I thought I was going to get the assassin killed. This was an evil campaign, remember, and part of the fun is plotting your friend's downfalls. In order to sell my story, though, I gave the assassin a knife I had gotten from the BBEG earlier... not realizing it was a MacGuffin the GM had planted, thinking it would be cool at the end of the campaign to use a token of friendship from the bad guy to kill him. Which it was! ...About a dozen sessions early. But did I let on that I was surprised? Hell no! As far as anyone knew, it was all part of the plan, and the fact that I had "accounted for it" made me look super devious.

Everything else about being devious you've already got - the abilities to plan and improvise are crucial. But remember, the party can't see your notes - they don't have to know you're improvising, or that you didn't have a response planned, or that what they just did wasn't part of the plan all along.

One last bit of advice, for dealing with the devious player, one that my GM learned a little late:

4. 
   Liars Should Always Assume They're Being Lied To - this is a terrible practice for dealing with real life people, but you're role-playing a villain. You don't have to see through every deception to be a good villain, but it helps make you look more devious if your plans assume that you can't trust anyone, even the players. As Princess Bride taught me, "I always assume everything is a trap. It's why I'm still alive."

Mainly I would say that it's great that you are not trying to force your schemes for the sake of "arcs" you had pre-planned, and that you allow players to figure out things if they can and play the game logically based on what happens. Running situations fairly and letting situations develop logically especially when players cleverly find ways to have unexpected successes, tends to make a game with scheming vastly more fun and interesting for the players, compared to games where GMs force things to go in pre-planned ways.

I would suggest that you persevere and develop some new techniques. If and when your NPCs do succeed in pulling off some unexpected schemes, it will be much more satisfying knowing that you do let them catch on to schemes if and when they can.

One technique that I have seen work wonders is to simply add a more detail to descriptions including observations, objects, characters, NPC behavior and remarks and so on that aren't particularly significant to anything important, or that are minor clues that would take a lot more investigation to make much out of. If you mainly only include details that are relevant to your planned schemes, it's much easier for players to perceive those schemes than if you also describe many other things that aren't relevant to those schemes, but are clues to various other things about the game world.

Details can also tend to make the world seem more like a real living place, and invite players to immerse and explore them.

Similarly, rolling more dice without explanation and consulting your notes but not saying what that's about, and/or passing private notes to players which may just be something their character notices, but that they should not show to other players, keeps players guessing and sometimes inspires them to pursue various schemes of their own.

Another technique that tends to work well, and is used in real life as well as Game of Thrones, is to organize schemes into "cells" where a schemer delegates to others who in turn delegate to others, with each level of the scheme hopefully only knowing what they need to in order to do their part, and possibly having misinformation. This allows some schemes to be discovered without exposing the larger schemes, allowing time to adjust and form new schemes.

Another interesting approach the GURPS Basic Set suggests (in Fourth Edition, on page 493) could be to use an Adversary player who is clever. i.e. Have another player who is playing some of the NPCs. It can also work well to simply find other clever people to consult for ideas even if they're not there during the game.

A devious character understands their goals, their potential resources, their potential obstacles, and the incentives of any potentially-involved parties.

A devious antagonist is always focused on their goals, but they aren't necessarily directly pursuing those goals at all times. Instead they work to build their advantages up when opportunities to do so arise, with every step making their goals easier to reach.

I usually have the opposite problem: enough of my villains have ultimately been revealed to be devious enough that my players tend to assume any suspicious-seeming NPC is up to something, even if they can't figure out what that something is.

An NPC that is actually devious isn't simply one that commits itself to a dishonest or obfuscated strategy. An NPC that is actually devious is one that is ready to recognize and seize non-obvious opportunities and constantly builds their advantage until conditions are favorable for striking at their goals.

Goals are what the NPC ultimately wants to happen, but that the NPC for some reason can't accomplish immediately. There is nothing special about the goals a devious character might have relative to any other type of character.

Potential obstacles are the reasons that the NPC's goal is still a goal rather than an accomplished fact. These are usually where most of the deviousness lies-- the NPC needs to avoid or overcome the obstacles to accomplish the goal, and for some reason cannot or prefers not to do so directly. A devious character has a good understanding of what elements lie between their current situation and their goals, and why they make the goals harder to accomplish.

Potential resources are things or situations that the NPC can use to overcome or avoid the obstacles that are preventing them from realizing their goals. A devious character has an intimate understanding of what resources they personally command, and is creative and subtle about recognizing non-obvious possible resources and matching them against obstacles.

Incentives of potentially-involved parties is the glue that makes a devious scheme hold together when other people or groups are in play, and a devious character will work to align others' incentives in such a way that people pursuing their own (perceived) self-interest will also move the devious character towards their goals.

A competent, devious NPC will always have an as-comprehensive-as-possible current accounting of those four elements, will at all times look for opportunities to connect them together, and will never choose to allow obstacles to worsen or resources to be depleted to no purpose.

Thinking through those elements is how I get into the mindset of a devious character, both for laying their overarching schemes and also for having them respond to new circumstances.

When you say

On several occasions, I have tried to have an NPC dupe my players into being pawns. And every time, one particular player has caught on and turned the tables; in one case even double-crossing the guy. I don't mind my players succeeding, but in every case that its happened, I had much larger arcs planned, and they were short-circuited.

my first thought is that a devious NPC should never reveal, or allow to be revealed, so much of their plan that a player can catch on, and that same NPC should have strategies in place so that a PC catching on doesn't expand the PC's list of options and won't endanger the NPC's goals.

So the NPC's plan shouldn't be to tell PCs to "go steal a MacGuffin from X", but then give directions to Y instead. That's too easy for the PCs to figure out, and the solution is too obvious (scheme against the NPC with Y). That's not a devious plan, it's a simple plan with exactly one straightforward twist.

A devious plan would recognize that the PCs are both potential resources and potential obstacles, and seek to arrange matters such that the PC's incentives push them ever further towards being better resources and less effective obstacles. If the NPC has the PCs do something that would turn Y against the PC party (steal an irrelevant item, interfere in some plan) before sending them to steal the MacGuffin, then:

• The PCs' value as resources doesn't change (they can still be sent to
  steal the MacGuffin from Y)


• The PCs' incentives are different than they were before (it'll be a
  lot harder for them to team up with Y, if Y doesn't trust them and
  starts working against them)


• Due to those different incentives, the PCs' threat as potential
  obstacles is reduced (it'll be harder for them to betray the NPC to
  Y, whether or not they discover the NPC's deceptions)


• The PCs have now been maneuvered in such a way their actions are more
  likely to bring the NPC closer to the NPC's goal and are less likely
  to effectively oppose the NPC

The specifics of how the NPC goes about this are mostly irrelevant. The deviousness is in working those four elements, and it doesn't matter if the NPC deceives the PCs about what they're doing, blackmails the PCs into knowingly doing what the NPC wants, or simply makes it unlikely and risky for the PCs to frustrate the schemes by collaborating with Y.

Manipulating any of those elements could be a minor story arc on its own, before the NPC sends the PCs to do anything that is even tangentially involved with their actual goals. And, crucially, the NPC will not willingly position the PCs anywhere remotely near their plots unless the NPC is affirmatively certain that the PCs can be reliably bent to serve the NPC's interests. The NPC will also keep tabs on the PCs to make sure they're not becoming more serious potential obstacles, and will be proactive in counteracting any movements in that direction.

tl;dr: A devious NPC will almost never set PCs on the trail of their actual goals, and will work to keep PCs from fully understanding the goals or the broader plans and situations around them. Instead the NPC will set PCs against obstacles the NPC has already perceived for other, plausible reasons, and will try to undermine the PCs' ability to act as obstacles themselves. Finally, the NPC will work to re-align incentives such that the PCs' self interest will at least seem to align with what the NPC wants, ideally to the extent that a PC figuring out what's really going on doesn't give them more scope or reason to do things that counteract the NPC's aims.

NPCs are exactly as devious as you want them to be

As the DM the players' entire worldview is coloured by how you describe it to them. If you want a character to be devious, they are. But there are few things you can do to stop the player's knowing that.

Showing your hand too early

I believe you are likely focusing so heavily on "I need this character to be devious" that you are accidentally giving the game away before you mean to. Maintaining the careful balance between giving the players enough clues that they can actually figure it out, but not so many that is it abundantly obvious is the difficult part of running characters like this.

There is a related question on hiding an evil character from the players until the big reveal. In my answer to that question I said:

Do evil people constantly do large evil things? No, they do small everyday things that add up to an evil whole. A clever villain knows not to show their hand too early.

The same advice applies here. These NPCs you are playing as clever and well-versed in hiding their devious ways. They aren't about to slip up and give away their entire plan just because the players asked nicely.

Use your DM powers to help the NPC

The other answers to this question (particularly by TheVagrantDog and Upper_Case) give good advice on how to play a devious character, but they have one major flaw. They forget that NPCs don't need to obey the rules of the game the way PCs do. By playing an NPC as though they were a PC you are forgetting one of the most powerful DM tools, and one that can make you seem far more devious than you really are.

You don't need to have a devious plan for your NPC to be devious. You can cheat. As DM you can literally reshape the world to make your evil plan work. If the PCs cause a problem with one part of your plan; oh well, guess that wasn't the plan in the first place, or there was a backup plan, or that ally or the party has secretly been spying on them the whole time. Remember: if you haven't told something to the players, it isn't real yet, you can change it.

Instead of pre-planning a complicated plot that the players are sure to uncover. You give your NPC a goal and allow the game to unfold until that goal can be realised. Then when you have your reveal it looks like that was your plan all along. Trust me, the players won't pick up on it (mine never have).

Your job as DM is to make the game fun, not to follow the rules. If you believe that this character being truly devious will make the story more enjoyable you changes things so that they can be devious. This isn't a license to change things behind your players backs and make it impossible to win, it is a tool to make the game better for everyone and you should use it.

Talk to that one player

You said:

And every time, one particular player has caught on and turned the tables; in one case even double-crossing the guy.

If this is only a problem because of one player, I suggest you talk to them outside of the session. Tell them how them being clever is making it harder for you to tell an interesting story and see if they want to help you out. Either you can get them to put a damper on their suspicious, or better you can get them to actively help you.

I have been in that position as a player. As an experienced DM playing in a new-DM's game, I could predict some of their moves and saw through several of their clever plots. Instead of exposing them and ruining the game for everyone, I talked to the DM between sessions, I pointed out where I had picked up on things and gave advice on how to improve. Additionally I avoided overdoing my own devious plans to make it easier for them to manage. As the campaign went on this became less necessary as them DM learned and improved, recently they pulled something on me that had me actively swearing at them (in a good way).

Ask your player if they can serve a similar role for you. Or get them involved in running one of the NPC from behind the scenes. If you can work together well it will make the game better for everyone.

I've been a game master for 20 years and let me tell you a secret: I suck at playing bad guys. Specially the ones that should be conniving, smart, and magnificently duplicitous.

So here are some tips for your schemes that may help you :)

Corporations (in late medieval to modern settings) and secret societies (in general) are two of my favourite staples, because you can define motivations just like a character, but the people are all cogs. Also they provide good campaign arcs where your players have to discover how to dismantle the operation without razing the entire town, and it even may end with them discovering that "you cut one head, but two will grow" etc etc. It's trite but effective.

I love, specially with new players, to make the villain the person they don't expect it to be. Examples: The helpful tavern owner -that is a shape-shifting monster disguised as a human, preying on the town-, the always sunny priest -who is really an agent of the god of sickness and is crafting a superspell using the town's prayers-, the boring mayor of the town -that is a slave of a lych, preparing the town for it's soon-to-hapen resurrection-, the town drunkard-that is just a disguise, meant to make easy to spy on you-. Just play them straight until they turn heel. This trick gets old fast so use it sparingly.

Misdirect: put an obvious antagonist: a universally hated sheriff -that simply is too strict for this lazy town-, an evil warlock that lives in a secluded cabin -he just wants to be left alone and has zero social skills-, the owner of the other tavern - who happens to be the only half-orc in a human town -, the always scheming clerk -he's just corrupt-evil, not child-eating-evil-, etc... Any figure of authority is always a good one because they usually have good reasons to keep interfering with you, specially if they think the players are the danger.

Presentation is everything! Players are usually wary of anything you mention in the room - I call it the "chekov's villain syndrome" :D - so don't stop only at the important things, but add all the details you can afford: if you are at an inn, describe some of the patrons loudly talking in a table, the bartender, the waitress - the things you would notice when entering a restaurant. In fact, I encourage you to go to places and take notes of what you notice about them; it makes descriptions more realistic. You risk them behaving a bit like the Knights of the Dinner table characters - who, at some point kept a cow as pet because it glowed magically, but usually some encouragement will keep them in check.

If you like crafting your own stories I suggest to take some writing and acting classes. I took some voice acting ones and the players reacted to it very positively.

Finally, if you can afford access to several player groups you can ask to "betatest" your characters. Some of my players - specially those who also are game directors - love "breaking my baddies" better than playing my campaigns, if only because of the laughs when they do something completely inappropriate. They also tend to offer good pointers and ways to improve your performance, and helps you anticipate some of your player's reactions.

An Example

Picture this: your heroes arrive to town in time to watch a public beating in the main plaza. A strongly built guy clad in a black and red uniform, is systematically punching a guy dressed in civilian clothes, that is being held by two brutish men in black clothes.

Most of the people in the plaza is watching the beating, but one guy, that was standing next to the Inn's door, approaches you. He's a balding, short and a bit overweight human, wearing a brown apron over simple undyed clothes - in short, a tavern-keeper.

-"New in town? I'm sorry for the welcome but the new town's Justice likes his spectacle" - players will notice he's trying to be polite but he's a bit nervous - "He's been in charge for several months now, but I cannot get used to it" - he cleans his hands again and offers you a handshake - "I'm Martin, the town's innkeeper. It's the best inn in town - if only because there is just this one!" - he laughs nervously.

Martin will politely inquire what are the motive of the group arrival, and how long they plan to stay. If asked about why the beating he will answer "Well, you know, this and that" - he lowers his voice almost to a whisper - "Master Rem seems to always find the excuse to fill the quota of beatings, if you catch my drift".

At some point, he will part with "Anyway, if you need food, drinks or lodging, come by my inn. I'll fix "

By the time the conversation ends, the beating has concluded and the town's Justice is peeling the padded leather gloves that was using for the beating, while his minions take the guy, who lost consciousness, back to the barracks. The Justice is watching the players with a mix of contempt and suspiciousness, but says nothing. If not approached or interrupted, he will follow his lackeys to the barracks.

When later, victims start appearing brutalized, who do you think the characters will suspect, the justice, or the innkeeper? And what if I told you the real baddy was the guy who was being beaten by the justice? cue dramatic chimpmunk

Hope this helps you a bit :)

Interestingly, I'm accustomed to the exact opposite. I'll build an innocent helpless NPC and the players will assume that it's the bad guy in disguise. They'll ask bunches of questions and try to catch them in a mistake. The sad part is when they do. No DM is perfect, and I hate to retcon things that happened last session...

My advice is to remember that the story hasn't been written beforehand. It's collaborative - and it's meant to be fun. If the players want this NPC to be good/evil, then think about it and decide what would be the most fun. Likewise, if you want a betrayal situation, then think about who the players DO trust - and then think about how that could be used against them.

Use sparingly or you'll end up in my predicament where my players don't trust anybody. Which brings up a good point. The players will be more/less suspicious depending on how often they get tricked. If almost all of your NPCs are the "good guys" - then the traitor is a needle in a haystack. If the opposite is true... the players will change according to what they have experienced.

You mentioned you're good at improvising, double down on that and have one large backup scheme (more on that one in a second).

If you're happy with the way your plans were foiled, except that a lot of your hard work went to waste, just don't do the hard work. Players (in my experience) like to feel smart and validated, and what better way to do that then appear to foil the BBEG at every turn?

My suggestion, though, is to have one big scheme working in the background.
I'm picturing something like magical gems that are found each time / most times when a plan is foiled, which the players can analyse and obsess over or just add to their loot bags and move on.
Then, when the time comes for the final encounter to play out, reveal that the gems were leeching knowledge from the party which will now fuel the BBEG's rune golem.

If they're still carrying the gemstones, have them dramatically float out to the golem. If they sold them, reveal that henchmen went to the shops and bought them back (and that the knowledge absorption was instantaneous), robbed the vault where they were stored, whatever. Hell, even if the party is super careful and actively destroys every gemstone, have them reformed from the aether in front of their noses and explain that the very act of them destroying them caused the knowledge absorption.

And if you're not really running a combat heavy game, have the gems tie into some other big plot that the BBEG is planning.

This way you'll have one big mystery on the back burner, which the party might be aware of, and a foiled plan will still feel like a minor victory to you as GM because you've set up another piece of the grand reveal with another gem presented to the party.

I have relatively little to add, but I'll imbue this bit of wisdom from John Lasseter of Pixar/Disney:

Using a coincidence to get your characters out of trouble is lazy. Using a coincidence to get them into trouble is good writing. Anything you like may go horribly wrong against the players to sort of railroad saving your bad guy as needed. You can get them out of the immediate danger after a rampaging bear comes out of nowhere and distracts the party, or attacks their beloved NPC.

Later on, you can retcon and figure out a reason that your BBEG might have some ability to commune with nature and control to local savage wildlife or something. Or you could have the city guard come in and grab the party on charges of murder, only for them to be dropped suspiciously quickly after the BBEG gets away. And just because your BBEG was outed once, it doesn't mean that they're outed forever. They can disguise themselves, they can make use of minions to do their dirty work, etc.