Auto replacement of charactersSubstitute a character by another one (Lualatex)XeTeX, change font usage when meeting greek characters (Sabon Linotype / Sabon Greek)Using active characters to swap fonts?Myanmar (Burmese) line breaking with XeLaTeXHow to input characters into XeLaTeXSetting glyph spacing with mathspecWhat is ‘mapping’ and how can I use it to my advantage?Calligraphy font with latin-extende-a alphabetPeek ahead and process charactersMarujirushi symbol won't display using XelatexXeLaTeX typesets characters in some Lato fonts on top of each other

Turing Machines: What is the difference between recognizing, deciding, total, accepting, rejecting?

Should I hide my travel history to the UK when I apply for an Australian visa?

Do human thoughts interact with matter?

Is there any way for a Half-Orc Sorcerer to get proficiency with a heavy weapon?

Should I cross-validate metrics that were not optimised?

My mother co-signed for my car. Can she take it away from me if I am the one making car payments?

Do I pay more tax when 2 extra payments are in this paypacket

Old story where computer expert digitally animates The Lord of the Rings

Why would a propellor have blades of different lengths?

How to widen the page

Why did moving the mouse cursor cause Windows 95 to run more quickly?

Birthday girl's casino game

HTTPmodule or similar for SharePoint online

Which high-degree derivatives play an essential role?

"Best practices" for formulating MIPs

Why is 一日 used instead of 日 to ask how your day is?

Contributing to a candidate as a Foreign National US Resident?

What is -(-2,3,4)?

Who are the police in Hong Kong?

How can I know (without going to the station) if RATP is offering the Anti Pollution tickets?

List of Implementations for common OR problems

Why did my leaking pool light trip the circuit breaker, but not the GFCI?

C++20 with u8, char8_t and std::string?

Are the plates of a battery really charged?



Auto replacement of characters


Substitute a character by another one (Lualatex)XeTeX, change font usage when meeting greek characters (Sabon Linotype / Sabon Greek)Using active characters to swap fonts?Myanmar (Burmese) line breaking with XeLaTeXHow to input characters into XeLaTeXSetting glyph spacing with mathspecWhat is ‘mapping’ and how can I use it to my advantage?Calligraphy font with latin-extende-a alphabetPeek ahead and process charactersMarujirushi symbol won't display using XelatexXeLaTeX typesets characters in some Lato fonts on top of each other






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;








4















To make it very simple, I want to change the mapping of keyboard keys and unicode characters, but not by programming. I want to do it with LaTeX.



eg. As suggested by @Ulrike and @Phelype I've edited my MWE.



This was the code provided to me by Ulrike.



documentclassarticle

directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["अ"] = "ə",
["ब"] = "b",
["क"] = "k",
["ड"] = "ɖ",
,



usepackagefontspec
setmainfontDoulos SIL%
[
RawFeature=+shuffle,
]


begindocument
अबकड
enddocument


Note - The script used here is Devanagari. The font specified is an SIL font. This kind of input will enable me to convert the data collected in local orthography to be processed in a phonetic script. The font which is used by fontspec is for the phonetic alphabet. I use XeLaTeX, so every character that I provide (with an appropriate font) looks exactly the same. I don't know how it happens in Lua.



PS - I know that my text editor can replace the characters, but I want to do it for full keyboard and not just for two or three characters. So it will be very tedious to do it for every character. Hence changing the mapping is possibly the best solution for my problem right now.










share|improve this question
























  • Are you open to using LuaLaTeX?

    – Mico
    8 hours ago






  • 1





    I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua

    – Niranjan
    8 hours ago






  • 1





    You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where 2pt would be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things like textitabc would not be converted. So try to tell us more about how this would actually be used.

    – Steven B. Segletes
    8 hours ago











  • As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e., myreplaceabcd which is rendered as pqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use the tr command in the terminal, or sed or awk or perl etc. etc. to replace everything with a single command.

    – Marijn
    8 hours ago






  • 1





    @Niranjan - Then you should be using Ulrike's answer.

    – Mico
    7 hours ago

















4















To make it very simple, I want to change the mapping of keyboard keys and unicode characters, but not by programming. I want to do it with LaTeX.



eg. As suggested by @Ulrike and @Phelype I've edited my MWE.



This was the code provided to me by Ulrike.



documentclassarticle

directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["अ"] = "ə",
["ब"] = "b",
["क"] = "k",
["ड"] = "ɖ",
,



usepackagefontspec
setmainfontDoulos SIL%
[
RawFeature=+shuffle,
]


begindocument
अबकड
enddocument


Note - The script used here is Devanagari. The font specified is an SIL font. This kind of input will enable me to convert the data collected in local orthography to be processed in a phonetic script. The font which is used by fontspec is for the phonetic alphabet. I use XeLaTeX, so every character that I provide (with an appropriate font) looks exactly the same. I don't know how it happens in Lua.



PS - I know that my text editor can replace the characters, but I want to do it for full keyboard and not just for two or three characters. So it will be very tedious to do it for every character. Hence changing the mapping is possibly the best solution for my problem right now.










share|improve this question
























  • Are you open to using LuaLaTeX?

    – Mico
    8 hours ago






  • 1





    I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua

    – Niranjan
    8 hours ago






  • 1





    You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where 2pt would be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things like textitabc would not be converted. So try to tell us more about how this would actually be used.

    – Steven B. Segletes
    8 hours ago











  • As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e., myreplaceabcd which is rendered as pqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use the tr command in the terminal, or sed or awk or perl etc. etc. to replace everything with a single command.

    – Marijn
    8 hours ago






  • 1





    @Niranjan - Then you should be using Ulrike's answer.

    – Mico
    7 hours ago













4












4








4


1






To make it very simple, I want to change the mapping of keyboard keys and unicode characters, but not by programming. I want to do it with LaTeX.



eg. As suggested by @Ulrike and @Phelype I've edited my MWE.



This was the code provided to me by Ulrike.



documentclassarticle

directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["अ"] = "ə",
["ब"] = "b",
["क"] = "k",
["ड"] = "ɖ",
,



usepackagefontspec
setmainfontDoulos SIL%
[
RawFeature=+shuffle,
]


begindocument
अबकड
enddocument


Note - The script used here is Devanagari. The font specified is an SIL font. This kind of input will enable me to convert the data collected in local orthography to be processed in a phonetic script. The font which is used by fontspec is for the phonetic alphabet. I use XeLaTeX, so every character that I provide (with an appropriate font) looks exactly the same. I don't know how it happens in Lua.



PS - I know that my text editor can replace the characters, but I want to do it for full keyboard and not just for two or three characters. So it will be very tedious to do it for every character. Hence changing the mapping is possibly the best solution for my problem right now.










share|improve this question
















To make it very simple, I want to change the mapping of keyboard keys and unicode characters, but not by programming. I want to do it with LaTeX.



eg. As suggested by @Ulrike and @Phelype I've edited my MWE.



This was the code provided to me by Ulrike.



documentclassarticle

directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["अ"] = "ə",
["ब"] = "b",
["क"] = "k",
["ड"] = "ɖ",
,



usepackagefontspec
setmainfontDoulos SIL%
[
RawFeature=+shuffle,
]


begindocument
अबकड
enddocument


Note - The script used here is Devanagari. The font specified is an SIL font. This kind of input will enable me to convert the data collected in local orthography to be processed in a phonetic script. The font which is used by fontspec is for the phonetic alphabet. I use XeLaTeX, so every character that I provide (with an appropriate font) looks exactly the same. I don't know how it happens in Lua.



PS - I know that my text editor can replace the characters, but I want to do it for full keyboard and not just for two or three characters. So it will be very tedious to do it for every character. Hence changing the mapping is possibly the best solution for my problem right now.







xetex lua






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 6 hours ago







Niranjan

















asked 8 hours ago









NiranjanNiranjan

1025 bronze badges




1025 bronze badges












  • Are you open to using LuaLaTeX?

    – Mico
    8 hours ago






  • 1





    I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua

    – Niranjan
    8 hours ago






  • 1





    You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where 2pt would be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things like textitabc would not be converted. So try to tell us more about how this would actually be used.

    – Steven B. Segletes
    8 hours ago











  • As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e., myreplaceabcd which is rendered as pqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use the tr command in the terminal, or sed or awk or perl etc. etc. to replace everything with a single command.

    – Marijn
    8 hours ago






  • 1





    @Niranjan - Then you should be using Ulrike's answer.

    – Mico
    7 hours ago

















  • Are you open to using LuaLaTeX?

    – Mico
    8 hours ago






  • 1





    I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua

    – Niranjan
    8 hours ago






  • 1





    You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where 2pt would be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things like textitabc would not be converted. So try to tell us more about how this would actually be used.

    – Steven B. Segletes
    8 hours ago











  • As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e., myreplaceabcd which is rendered as pqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use the tr command in the terminal, or sed or awk or perl etc. etc. to replace everything with a single command.

    – Marijn
    8 hours ago






  • 1





    @Niranjan - Then you should be using Ulrike's answer.

    – Mico
    7 hours ago
















Are you open to using LuaLaTeX?

– Mico
8 hours ago





Are you open to using LuaLaTeX?

– Mico
8 hours ago




1




1





I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua

– Niranjan
8 hours ago





I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua

– Niranjan
8 hours ago




1




1





You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where 2pt would be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things like textitabc would not be converted. So try to tell us more about how this would actually be used.

– Steven B. Segletes
8 hours ago





You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where 2pt would be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things like textitabc would not be converted. So try to tell us more about how this would actually be used.

– Steven B. Segletes
8 hours ago













As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e., myreplaceabcd which is rendered as pqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use the tr command in the terminal, or sed or awk or perl etc. etc. to replace everything with a single command.

– Marijn
8 hours ago





As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e., myreplaceabcd which is rendered as pqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use the tr command in the terminal, or sed or awk or perl etc. etc. to replace everything with a single command.

– Marijn
8 hours ago




1




1





@Niranjan - Then you should be using Ulrike's answer.

– Mico
7 hours ago





@Niranjan - Then you should be using Ulrike's answer.

– Mico
7 hours ago










3 Answers
3






active

oldest

votes


















8














with lualatex:



documentclassarticle
directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,



usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]


begindocument
abcd
enddocument


enter image description here






share|improve this answer























  • Alternatively you can use type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.

    – Marijn
    7 hours ago











  • Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph

    – Niranjan
    7 hours ago






  • 1





    well if directlua is not defined that you didn't use lualatex (but probably xelatex).

    – Ulrike Fischer
    7 hours ago











  • @Niranjan if your question is about transliteration: with xelatex you could also use a teckit mapping. Simply search the site, you will find a number of answers.

    – Ulrike Fischer
    7 hours ago











  • Please see the edit.

    – Niranjan
    7 hours ago


















4














OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.



The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.



The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.



So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:



deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS


It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.



So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of



tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR


At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:



tokcycle[]<input stream>

expandedtokcycle[]<input stream>

begintokencycle[]
<input stream>
endtokencycle

beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle


The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).



Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.



Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).



Now for the code. First, the code to address the OP's problem:



documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS

begindocument
%अबकड

verb|tcremap| handles a single token: tcremapअ.

texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.

noindenthrulefill

ENVIRONMENT, DIRECT APPROACH

begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro

अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.
endtokencycle

noindenthrulefill

MACRO, TOKENIZED APPROACH

cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.

toksrescancytoks
thecytoks
enddocument


enter image description here




And for the tokcycle code, which I hope to officially release in the next few weeks:



First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.



% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11

% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle

def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%

longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%


def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone

% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next

% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR

% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg

deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp

deftc@resetmacrolet@tcusemacrotc@empty

% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg

% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1

defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%


% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks

%--------------------------------

input listofitems.tex

% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12

% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%


% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi


catcode`@=12
endinput


Then there is the LaTeX style wrapper, tokcycle.sty:



ProvidesPackagetokcycle[V0.9]

usepackageenviron

%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR

% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR

% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR

% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR

input tokcycle.tex

endinput


Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.






share|improve this answer

























  • Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.

    – Sebastiano
    3 hours ago






  • 2





    @Sebastiano I've been working on this about 5 weeks.

    – Steven B. Segletes
    3 hours ago











  • Impressive :-(. My best compliments.

    – Sebastiano
    3 hours ago


















1














Really, really bad idea :-)



You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.



I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:




enter image description here




documentclassarticle

newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup

begindocument

`abcd'remapchars`abcd'`abcd'

enddocument





share|improve this answer























  • It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?

    – Niranjan
    7 hours ago











  • @Niranjan Sorry, I have no idea about Devanagari. I did a test remapping देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?

    – Phelype Oleinik
    7 hours ago












  • @Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)

    – Phelype Oleinik
    7 hours ago












  • Oh ok. Let me add the exact input and output in my question.

    – Niranjan
    7 hours ago











  • Please see the edit

    – Niranjan
    7 hours ago













Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "85"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2ftex.stackexchange.com%2fquestions%2f498243%2fauto-replacement-of-characters%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























3 Answers
3






active

oldest

votes








3 Answers
3






active

oldest

votes









active

oldest

votes






active

oldest

votes









8














with lualatex:



documentclassarticle
directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,



usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]


begindocument
abcd
enddocument


enter image description here






share|improve this answer























  • Alternatively you can use type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.

    – Marijn
    7 hours ago











  • Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph

    – Niranjan
    7 hours ago






  • 1





    well if directlua is not defined that you didn't use lualatex (but probably xelatex).

    – Ulrike Fischer
    7 hours ago











  • @Niranjan if your question is about transliteration: with xelatex you could also use a teckit mapping. Simply search the site, you will find a number of answers.

    – Ulrike Fischer
    7 hours ago











  • Please see the edit.

    – Niranjan
    7 hours ago















8














with lualatex:



documentclassarticle
directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,



usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]


begindocument
abcd
enddocument


enter image description here






share|improve this answer























  • Alternatively you can use type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.

    – Marijn
    7 hours ago











  • Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph

    – Niranjan
    7 hours ago






  • 1





    well if directlua is not defined that you didn't use lualatex (but probably xelatex).

    – Ulrike Fischer
    7 hours ago











  • @Niranjan if your question is about transliteration: with xelatex you could also use a teckit mapping. Simply search the site, you will find a number of answers.

    – Ulrike Fischer
    7 hours ago











  • Please see the edit.

    – Niranjan
    7 hours ago













8












8








8







with lualatex:



documentclassarticle
directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,



usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]


begindocument
abcd
enddocument


enter image description here






share|improve this answer













with lualatex:



documentclassarticle
directlua

fonts.handlers.otf.addfeature

name = "shuffle",
type = "multiple",
data =

["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,



usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]


begindocument
abcd
enddocument


enter image description here







share|improve this answer












share|improve this answer



share|improve this answer










answered 8 hours ago









Ulrike FischerUlrike Fischer

207k9 gold badges314 silver badges708 bronze badges




207k9 gold badges314 silver badges708 bronze badges












  • Alternatively you can use type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.

    – Marijn
    7 hours ago











  • Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph

    – Niranjan
    7 hours ago






  • 1





    well if directlua is not defined that you didn't use lualatex (but probably xelatex).

    – Ulrike Fischer
    7 hours ago











  • @Niranjan if your question is about transliteration: with xelatex you could also use a teckit mapping. Simply search the site, you will find a number of answers.

    – Ulrike Fischer
    7 hours ago











  • Please see the edit.

    – Niranjan
    7 hours ago

















  • Alternatively you can use type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.

    – Marijn
    7 hours ago











  • Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph

    – Niranjan
    7 hours ago






  • 1





    well if directlua is not defined that you didn't use lualatex (but probably xelatex).

    – Ulrike Fischer
    7 hours ago











  • @Niranjan if your question is about transliteration: with xelatex you could also use a teckit mapping. Simply search the site, you will find a number of answers.

    – Ulrike Fischer
    7 hours ago











  • Please see the edit.

    – Niranjan
    7 hours ago
















Alternatively you can use type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.

– Marijn
7 hours ago





Alternatively you can use type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.

– Marijn
7 hours ago













Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph

– Niranjan
7 hours ago





Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph

– Niranjan
7 hours ago




1




1





well if directlua is not defined that you didn't use lualatex (but probably xelatex).

– Ulrike Fischer
7 hours ago





well if directlua is not defined that you didn't use lualatex (but probably xelatex).

– Ulrike Fischer
7 hours ago













@Niranjan if your question is about transliteration: with xelatex you could also use a teckit mapping. Simply search the site, you will find a number of answers.

– Ulrike Fischer
7 hours ago





@Niranjan if your question is about transliteration: with xelatex you could also use a teckit mapping. Simply search the site, you will find a number of answers.

– Ulrike Fischer
7 hours ago













Please see the edit.

– Niranjan
7 hours ago





Please see the edit.

– Niranjan
7 hours ago













4














OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.



The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.



The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.



So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:



deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS


It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.



So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of



tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR


At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:



tokcycle[]<input stream>

expandedtokcycle[]<input stream>

begintokencycle[]
<input stream>
endtokencycle

beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle


The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).



Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.



Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).



Now for the code. First, the code to address the OP's problem:



documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS

begindocument
%अबकड

verb|tcremap| handles a single token: tcremapअ.

texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.

noindenthrulefill

ENVIRONMENT, DIRECT APPROACH

begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro

अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.
endtokencycle

noindenthrulefill

MACRO, TOKENIZED APPROACH

cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.

toksrescancytoks
thecytoks
enddocument


enter image description here




And for the tokcycle code, which I hope to officially release in the next few weeks:



First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.



% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11

% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle

def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%

longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%


def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone

% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next

% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR

% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg

deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp

deftc@resetmacrolet@tcusemacrotc@empty

% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg

% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1

defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%


% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks

%--------------------------------

input listofitems.tex

% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12

% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%


% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi


catcode`@=12
endinput


Then there is the LaTeX style wrapper, tokcycle.sty:



ProvidesPackagetokcycle[V0.9]

usepackageenviron

%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR

% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR

% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR

% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR

input tokcycle.tex

endinput


Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.






share|improve this answer

























  • Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.

    – Sebastiano
    3 hours ago






  • 2





    @Sebastiano I've been working on this about 5 weeks.

    – Steven B. Segletes
    3 hours ago











  • Impressive :-(. My best compliments.

    – Sebastiano
    3 hours ago















4














OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.



The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.



The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.



So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:



deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS


It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.



So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of



tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR


At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:



tokcycle[]<input stream>

expandedtokcycle[]<input stream>

begintokencycle[]
<input stream>
endtokencycle

beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle


The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).



Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.



Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).



Now for the code. First, the code to address the OP's problem:



documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS

begindocument
%अबकड

verb|tcremap| handles a single token: tcremapअ.

texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.

noindenthrulefill

ENVIRONMENT, DIRECT APPROACH

begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro

अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.
endtokencycle

noindenthrulefill

MACRO, TOKENIZED APPROACH

cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.

toksrescancytoks
thecytoks
enddocument


enter image description here




And for the tokcycle code, which I hope to officially release in the next few weeks:



First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.



% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11

% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle

def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%

longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%


def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone

% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next

% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR

% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg

deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp

deftc@resetmacrolet@tcusemacrotc@empty

% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg

% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1

defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%


% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks

%--------------------------------

input listofitems.tex

% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12

% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%


% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi


catcode`@=12
endinput


Then there is the LaTeX style wrapper, tokcycle.sty:



ProvidesPackagetokcycle[V0.9]

usepackageenviron

%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR

% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR

% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR

% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR

input tokcycle.tex

endinput


Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.






share|improve this answer

























  • Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.

    – Sebastiano
    3 hours ago






  • 2





    @Sebastiano I've been working on this about 5 weeks.

    – Steven B. Segletes
    3 hours ago











  • Impressive :-(. My best compliments.

    – Sebastiano
    3 hours ago













4












4








4







OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.



The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.



The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.



So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:



deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS


It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.



So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of



tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR


At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:



tokcycle[]<input stream>

expandedtokcycle[]<input stream>

begintokencycle[]
<input stream>
endtokencycle

beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle


The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).



Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.



Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).



Now for the code. First, the code to address the OP's problem:



documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS

begindocument
%अबकड

verb|tcremap| handles a single token: tcremapअ.

texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.

noindenthrulefill

ENVIRONMENT, DIRECT APPROACH

begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro

अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.
endtokencycle

noindenthrulefill

MACRO, TOKENIZED APPROACH

cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.

toksrescancytoks
thecytoks
enddocument


enter image description here




And for the tokcycle code, which I hope to officially release in the next few weeks:



First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.



% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11

% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle

def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%

longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%


def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone

% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next

% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR

% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg

deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp

deftc@resetmacrolet@tcusemacrotc@empty

% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg

% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1

defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%


% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks

%--------------------------------

input listofitems.tex

% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12

% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%


% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi


catcode`@=12
endinput


Then there is the LaTeX style wrapper, tokcycle.sty:



ProvidesPackagetokcycle[V0.9]

usepackageenviron

%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR

% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR

% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR

% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR

input tokcycle.tex

endinput


Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.






share|improve this answer















OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.



The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.



The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.



So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:



deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS


It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.



So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of



tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR


At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:



tokcycle[]<input stream>

expandedtokcycle[]<input stream>

begintokencycle[]
<input stream>
endtokencycle

beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle


The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).



Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.



Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).



Now for the code. First, the code to address the OP's problem:



documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1

tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS

begindocument
%अबकड

verb|tcremap| handles a single token: tcremapअ.

texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.

noindenthrulefill

ENVIRONMENT, DIRECT APPROACH

begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro

अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.
endtokencycle

noindenthrulefill

MACRO, TOKENIZED APPROACH

cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.

Other text for which no mapping is yet given as of today.

अबक done.

toksrescancytoks
thecytoks
enddocument


enter image description here




And for the tokcycle code, which I hope to officially release in the next few weeks:



First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.



% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11

% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle

def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%

longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%


def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone

% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next

% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR

% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg

deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp

deftc@resetmacrolet@tcusemacrotc@empty

% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg

% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1

defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%


% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks

%--------------------------------

input listofitems.tex

% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12

% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%


% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi


catcode`@=12
endinput


Then there is the LaTeX style wrapper, tokcycle.sty:



ProvidesPackagetokcycle[V0.9]

usepackageenviron

%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR

% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR

% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR

% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR

input tokcycle.tex

endinput


Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.







share|improve this answer














share|improve this answer



share|improve this answer








edited 3 hours ago

























answered 4 hours ago









Steven B. SegletesSteven B. Segletes

167k9 gold badges210 silver badges429 bronze badges




167k9 gold badges210 silver badges429 bronze badges












  • Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.

    – Sebastiano
    3 hours ago






  • 2





    @Sebastiano I've been working on this about 5 weeks.

    – Steven B. Segletes
    3 hours ago











  • Impressive :-(. My best compliments.

    – Sebastiano
    3 hours ago

















  • Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.

    – Sebastiano
    3 hours ago






  • 2





    @Sebastiano I've been working on this about 5 weeks.

    – Steven B. Segletes
    3 hours ago











  • Impressive :-(. My best compliments.

    – Sebastiano
    3 hours ago
















Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.

– Sebastiano
3 hours ago





Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.

– Sebastiano
3 hours ago




2




2





@Sebastiano I've been working on this about 5 weeks.

– Steven B. Segletes
3 hours ago





@Sebastiano I've been working on this about 5 weeks.

– Steven B. Segletes
3 hours ago













Impressive :-(. My best compliments.

– Sebastiano
3 hours ago





Impressive :-(. My best compliments.

– Sebastiano
3 hours ago











1














Really, really bad idea :-)



You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.



I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:




enter image description here




documentclassarticle

newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup

begindocument

`abcd'remapchars`abcd'`abcd'

enddocument





share|improve this answer























  • It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?

    – Niranjan
    7 hours ago











  • @Niranjan Sorry, I have no idea about Devanagari. I did a test remapping देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?

    – Phelype Oleinik
    7 hours ago












  • @Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)

    – Phelype Oleinik
    7 hours ago












  • Oh ok. Let me add the exact input and output in my question.

    – Niranjan
    7 hours ago











  • Please see the edit

    – Niranjan
    7 hours ago















1














Really, really bad idea :-)



You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.



I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:




enter image description here




documentclassarticle

newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup

begindocument

`abcd'remapchars`abcd'`abcd'

enddocument





share|improve this answer























  • It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?

    – Niranjan
    7 hours ago











  • @Niranjan Sorry, I have no idea about Devanagari. I did a test remapping देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?

    – Phelype Oleinik
    7 hours ago












  • @Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)

    – Phelype Oleinik
    7 hours ago












  • Oh ok. Let me add the exact input and output in my question.

    – Niranjan
    7 hours ago











  • Please see the edit

    – Niranjan
    7 hours ago













1












1








1







Really, really bad idea :-)



You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.



I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:




enter image description here




documentclassarticle

newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup

begindocument

`abcd'remapchars`abcd'`abcd'

enddocument





share|improve this answer













Really, really bad idea :-)



You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.



I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:




enter image description here




documentclassarticle

newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup

begindocument

`abcd'remapchars`abcd'`abcd'

enddocument






share|improve this answer












share|improve this answer



share|improve this answer










answered 8 hours ago









Phelype OleinikPhelype Oleinik

30.9k7 gold badges52 silver badges105 bronze badges




30.9k7 gold badges52 silver badges105 bronze badges












  • It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?

    – Niranjan
    7 hours ago











  • @Niranjan Sorry, I have no idea about Devanagari. I did a test remapping देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?

    – Phelype Oleinik
    7 hours ago












  • @Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)

    – Phelype Oleinik
    7 hours ago












  • Oh ok. Let me add the exact input and output in my question.

    – Niranjan
    7 hours ago











  • Please see the edit

    – Niranjan
    7 hours ago

















  • It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?

    – Niranjan
    7 hours ago











  • @Niranjan Sorry, I have no idea about Devanagari. I did a test remapping देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?

    – Phelype Oleinik
    7 hours ago












  • @Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)

    – Phelype Oleinik
    7 hours ago












  • Oh ok. Let me add the exact input and output in my question.

    – Niranjan
    7 hours ago











  • Please see the edit

    – Niranjan
    7 hours ago
















It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?

– Niranjan
7 hours ago





It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?

– Niranjan
7 hours ago













@Niranjan Sorry, I have no idea about Devanagari. I did a test remapping देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?

– Phelype Oleinik
7 hours ago






@Niranjan Sorry, I have no idea about Devanagari. I did a test remapping देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?

– Phelype Oleinik
7 hours ago














@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)

– Phelype Oleinik
7 hours ago






@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)

– Phelype Oleinik
7 hours ago














Oh ok. Let me add the exact input and output in my question.

– Niranjan
7 hours ago





Oh ok. Let me add the exact input and output in my question.

– Niranjan
7 hours ago













Please see the edit

– Niranjan
7 hours ago





Please see the edit

– Niranjan
7 hours ago

















draft saved

draft discarded
















































Thanks for contributing an answer to TeX - LaTeX Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2ftex.stackexchange.com%2fquestions%2f498243%2fauto-replacement-of-characters%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Canceling a color specificationRandomly assigning color to Graphics3D objects?Default color for Filling in Mathematica 9Coloring specific elements of sets with a prime modified order in an array plotHow to pick a color differing significantly from the colors already in a given color list?Detection of the text colorColor numbers based on their valueCan color schemes for use with ColorData include opacity specification?My dynamic color schemes

Invision Community Contents History See also References External links Navigation menuProprietaryinvisioncommunity.comIPS Community ForumsIPS Community Forumsthis blog entry"License Changes, IP.Board 3.4, and the Future""Interview -- Matt Mecham of Ibforums""CEO Invision Power Board, Matt Mecham Is a Liar, Thief!"IPB License Explanation 1.3, 1.3.1, 2.0, and 2.1ArchivedSecurity Fixes, Updates And Enhancements For IPB 1.3.1Archived"New Demo Accounts - Invision Power Services"the original"New Default Skin"the original"Invision Power Board 3.0.0 and Applications Released"the original"Archived copy"the original"Perpetual licenses being done away with""Release Notes - Invision Power Services""Introducing: IPS Community Suite 4!"Invision Community Release Notes

François Viète Contents Biography Work and thought Bibliography See also Notes Further reading External links Navigation menup. 21Google Bookspp. 75–77Google BooksDe thou (from University of Saint Andrews)ArchivedGoogle BooksGoogle BooksGoogle BooksGoogle booksGoogle Bookscc-parthenay.frL'histoire universelle (fr)Universal History (en)ArchivedAdsabs.harvard.eduPagesperso-orange.frArchive.orgChikara Sasaki. Descartes' mathematical thought p.259Google BooksGoogle BooksGoogle Bookspp. 152 and onwardGoogle BooksGoogle BooksScribd.comGoogle Books1257-7979Google BooksGoogle BooksGoogle BooksGoogle BooksGoogle BooksGoogle BooksGallica.bnf.frGoogle BooksGoogle Books"François Viète"Francois Viète: Father of Modern Algebraic NotationThe Lawyer and the GamblerAbout TarporleySite de Jean-Paul GuichardL'algèbre nouvelle"About the Harmonicon"cb120511976(data)1188044800000 0001 0913 5903n82164680ola2013766880073431702w6vt1sb70287374827140948071409480