Auto replacement of charactersSubstitute a character by another one (Lualatex)XeTeX, change font usage when meeting greek characters (Sabon Linotype / Sabon Greek)Using active characters to swap fonts?Myanmar (Burmese) line breaking with XeLaTeXHow to input characters into XeLaTeXSetting glyph spacing with mathspecWhat is ‘mapping’ and how can I use it to my advantage?Calligraphy font with latin-extende-a alphabetPeek ahead and process charactersMarujirushi symbol won't display using XelatexXeLaTeX typesets characters in some Lato fonts on top of each other
Turing Machines: What is the difference between recognizing, deciding, total, accepting, rejecting?
Should I hide my travel history to the UK when I apply for an Australian visa?
Do human thoughts interact with matter?
Is there any way for a Half-Orc Sorcerer to get proficiency with a heavy weapon?
Should I cross-validate metrics that were not optimised?
My mother co-signed for my car. Can she take it away from me if I am the one making car payments?
Do I pay more tax when 2 extra payments are in this paypacket
Old story where computer expert digitally animates The Lord of the Rings
Why would a propellor have blades of different lengths?
How to widen the page
Why did moving the mouse cursor cause Windows 95 to run more quickly?
Birthday girl's casino game
HTTPmodule or similar for SharePoint online
Which high-degree derivatives play an essential role?
"Best practices" for formulating MIPs
Why is 一日 used instead of 日 to ask how your day is?
Contributing to a candidate as a Foreign National US Resident?
What is -(-2,3,4)?
Who are the police in Hong Kong?
How can I know (without going to the station) if RATP is offering the Anti Pollution tickets?
List of Implementations for common OR problems
Why did my leaking pool light trip the circuit breaker, but not the GFCI?
C++20 with u8, char8_t and std::string?
Are the plates of a battery really charged?
Auto replacement of characters
Substitute a character by another one (Lualatex)XeTeX, change font usage when meeting greek characters (Sabon Linotype / Sabon Greek)Using active characters to swap fonts?Myanmar (Burmese) line breaking with XeLaTeXHow to input characters into XeLaTeXSetting glyph spacing with mathspecWhat is ‘mapping’ and how can I use it to my advantage?Calligraphy font with latin-extende-a alphabetPeek ahead and process charactersMarujirushi symbol won't display using XelatexXeLaTeX typesets characters in some Lato fonts on top of each other
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
To make it very simple, I want to change the mapping of keyboard keys and unicode characters, but not by programming. I want to do it with LaTeX.
eg. As suggested by @Ulrike and @Phelype I've edited my MWE.
This was the code provided to me by Ulrike.
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["अ"] = "ə",
["ब"] = "b",
["क"] = "k",
["ड"] = "ɖ",
,
usepackagefontspec
setmainfontDoulos SIL%
[
RawFeature=+shuffle,
]
begindocument
अबकड
enddocument
Note - The script used here is Devanagari. The font specified is an SIL font. This kind of input will enable me to convert the data collected in local orthography to be processed in a phonetic script. The font which is used by fontspec is for the phonetic alphabet. I use XeLaTeX, so every character that I provide (with an appropriate font) looks exactly the same. I don't know how it happens in Lua.
PS - I know that my text editor can replace the characters, but I want to do it for full keyboard and not just for two or three characters. So it will be very tedious to do it for every character. Hence changing the mapping is possibly the best solution for my problem right now.
xetex lua
|
show 6 more comments
To make it very simple, I want to change the mapping of keyboard keys and unicode characters, but not by programming. I want to do it with LaTeX.
eg. As suggested by @Ulrike and @Phelype I've edited my MWE.
This was the code provided to me by Ulrike.
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["अ"] = "ə",
["ब"] = "b",
["क"] = "k",
["ड"] = "ɖ",
,
usepackagefontspec
setmainfontDoulos SIL%
[
RawFeature=+shuffle,
]
begindocument
अबकड
enddocument
Note - The script used here is Devanagari. The font specified is an SIL font. This kind of input will enable me to convert the data collected in local orthography to be processed in a phonetic script. The font which is used by fontspec is for the phonetic alphabet. I use XeLaTeX, so every character that I provide (with an appropriate font) looks exactly the same. I don't know how it happens in Lua.
PS - I know that my text editor can replace the characters, but I want to do it for full keyboard and not just for two or three characters. So it will be very tedious to do it for every character. Hence changing the mapping is possibly the best solution for my problem right now.
xetex lua
Are you open to using LuaLaTeX?
– Mico
8 hours ago
1
I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua
– Niranjan
8 hours ago
1
You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where2ptwould be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things liketextitabcwould not be converted. So try to tell us more about how this would actually be used.
– Steven B. Segletes
8 hours ago
As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e.,myreplaceabcdwhich is rendered aspqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use thetrcommand in the terminal, orsedorawkorperletc. etc. to replace everything with a single command.
– Marijn
8 hours ago
1
@Niranjan - Then you should be using Ulrike's answer.
– Mico
7 hours ago
|
show 6 more comments
To make it very simple, I want to change the mapping of keyboard keys and unicode characters, but not by programming. I want to do it with LaTeX.
eg. As suggested by @Ulrike and @Phelype I've edited my MWE.
This was the code provided to me by Ulrike.
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["अ"] = "ə",
["ब"] = "b",
["क"] = "k",
["ड"] = "ɖ",
,
usepackagefontspec
setmainfontDoulos SIL%
[
RawFeature=+shuffle,
]
begindocument
अबकड
enddocument
Note - The script used here is Devanagari. The font specified is an SIL font. This kind of input will enable me to convert the data collected in local orthography to be processed in a phonetic script. The font which is used by fontspec is for the phonetic alphabet. I use XeLaTeX, so every character that I provide (with an appropriate font) looks exactly the same. I don't know how it happens in Lua.
PS - I know that my text editor can replace the characters, but I want to do it for full keyboard and not just for two or three characters. So it will be very tedious to do it for every character. Hence changing the mapping is possibly the best solution for my problem right now.
xetex lua
To make it very simple, I want to change the mapping of keyboard keys and unicode characters, but not by programming. I want to do it with LaTeX.
eg. As suggested by @Ulrike and @Phelype I've edited my MWE.
This was the code provided to me by Ulrike.
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["अ"] = "ə",
["ब"] = "b",
["क"] = "k",
["ड"] = "ɖ",
,
usepackagefontspec
setmainfontDoulos SIL%
[
RawFeature=+shuffle,
]
begindocument
अबकड
enddocument
Note - The script used here is Devanagari. The font specified is an SIL font. This kind of input will enable me to convert the data collected in local orthography to be processed in a phonetic script. The font which is used by fontspec is for the phonetic alphabet. I use XeLaTeX, so every character that I provide (with an appropriate font) looks exactly the same. I don't know how it happens in Lua.
PS - I know that my text editor can replace the characters, but I want to do it for full keyboard and not just for two or three characters. So it will be very tedious to do it for every character. Hence changing the mapping is possibly the best solution for my problem right now.
xetex lua
xetex lua
edited 6 hours ago
Niranjan
asked 8 hours ago
NiranjanNiranjan
1025 bronze badges
1025 bronze badges
Are you open to using LuaLaTeX?
– Mico
8 hours ago
1
I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua
– Niranjan
8 hours ago
1
You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where2ptwould be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things liketextitabcwould not be converted. So try to tell us more about how this would actually be used.
– Steven B. Segletes
8 hours ago
As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e.,myreplaceabcdwhich is rendered aspqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use thetrcommand in the terminal, orsedorawkorperletc. etc. to replace everything with a single command.
– Marijn
8 hours ago
1
@Niranjan - Then you should be using Ulrike's answer.
– Mico
7 hours ago
|
show 6 more comments
Are you open to using LuaLaTeX?
– Mico
8 hours ago
1
I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua
– Niranjan
8 hours ago
1
You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where2ptwould be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things liketextitabcwould not be converted. So try to tell us more about how this would actually be used.
– Steven B. Segletes
8 hours ago
As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e.,myreplaceabcdwhich is rendered aspqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use thetrcommand in the terminal, orsedorawkorperletc. etc. to replace everything with a single command.
– Marijn
8 hours ago
1
@Niranjan - Then you should be using Ulrike's answer.
– Mico
7 hours ago
Are you open to using LuaLaTeX?
– Mico
8 hours ago
Are you open to using LuaLaTeX?
– Mico
8 hours ago
1
1
I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua
– Niranjan
8 hours ago
I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua
– Niranjan
8 hours ago
1
1
You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where
2pt would be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things like textitabc would not be converted. So try to tell us more about how this would actually be used.– Steven B. Segletes
8 hours ago
You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where
2pt would be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things like textitabc would not be converted. So try to tell us more about how this would actually be used.– Steven B. Segletes
8 hours ago
As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e.,
myreplaceabcd which is rendered as pqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use the tr command in the terminal, or sed or awk or perl etc. etc. to replace everything with a single command.– Marijn
8 hours ago
As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e.,
myreplaceabcd which is rendered as pqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use the tr command in the terminal, or sed or awk or perl etc. etc. to replace everything with a single command.– Marijn
8 hours ago
1
1
@Niranjan - Then you should be using Ulrike's answer.
– Mico
7 hours ago
@Niranjan - Then you should be using Ulrike's answer.
– Mico
7 hours ago
|
show 6 more comments
3 Answers
3
active
oldest
votes
with lualatex:
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,
usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]
begindocument
abcd
enddocument

Alternatively you can usetype=substitutionas mentioned in thefontspecmanual (page 57) - I'm not sure if there is any difference.
– Marijn
7 hours ago
Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph
– Niranjan
7 hours ago
1
well if directlua is not defined that you didn't use lualatex (but probably xelatex).
– Ulrike Fischer
7 hours ago
@Niranjan if your question is about transliteration: with xelatex you could also use ateckitmapping. Simply search the site, you will find a number of answers.
– Ulrike Fischer
7 hours ago
Please see the edit.
– Niranjan
7 hours ago
add a comment |
OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.
The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.
The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.
So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.
So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of
tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR
At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:
tokcycle[]<input stream>
expandedtokcycle[]<input stream>
begintokencycle[]
<input stream>
endtokencycle
beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle
The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).
Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.
Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).
Now for the code. First, the code to address the OP's problem:
documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
begindocument
%अबकड
verb|tcremap| handles a single token: tcremapअ.
texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.
noindenthrulefill
ENVIRONMENT, DIRECT APPROACH
begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
endtokencycle
noindenthrulefill
MACRO, TOKENIZED APPROACH
cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
toksrescancytoks
thecytoks
enddocument

And for the tokcycle code, which I hope to officially release in the next few weeks:
First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.
% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11
% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle
def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%
longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%
def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone
% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next
% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR
% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg
deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp
deftc@resetmacrolet@tcusemacrotc@empty
% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg
% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1
defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%
% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks
%--------------------------------
input listofitems.tex
% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12
% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%
% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi
catcode`@=12
endinput
Then there is the LaTeX style wrapper, tokcycle.sty:
ProvidesPackagetokcycle[V0.9]
usepackageenviron
%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR
% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR
% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR
% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR
input tokcycle.tex
endinput
Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.
Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.
– Sebastiano
3 hours ago
2
@Sebastiano I've been working on this about 5 weeks.
– Steven B. Segletes
3 hours ago
Impressive :-(. My best compliments.
– Sebastiano
3 hours ago
add a comment |
Really, really bad idea :-)
You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.
I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:
documentclassarticle
newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup
begindocument
`abcd'remapchars`abcd'`abcd'
enddocument
It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?
– Niranjan
7 hours ago
@Niranjan Sorry, I have no idea about Devanagari. I did a test remappingदेp,वq,नाr,गs, andरीt, and wroteremapchars`देवनागरी'(test file) but it came out asp r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?
– Phelype Oleinik
7 hours ago
@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. Withremap दp remap वq remap नr remap गs remap रtandremapchars`दवनगर'it works. (I think I've probably just slaughtered the Devanagari script...)
– Phelype Oleinik
7 hours ago
Oh ok. Let me add the exact input and output in my question.
– Niranjan
7 hours ago
Please see the edit
– Niranjan
7 hours ago
|
show 1 more comment
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "85"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2ftex.stackexchange.com%2fquestions%2f498243%2fauto-replacement-of-characters%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
with lualatex:
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,
usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]
begindocument
abcd
enddocument

Alternatively you can usetype=substitutionas mentioned in thefontspecmanual (page 57) - I'm not sure if there is any difference.
– Marijn
7 hours ago
Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph
– Niranjan
7 hours ago
1
well if directlua is not defined that you didn't use lualatex (but probably xelatex).
– Ulrike Fischer
7 hours ago
@Niranjan if your question is about transliteration: with xelatex you could also use ateckitmapping. Simply search the site, you will find a number of answers.
– Ulrike Fischer
7 hours ago
Please see the edit.
– Niranjan
7 hours ago
add a comment |
with lualatex:
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,
usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]
begindocument
abcd
enddocument

Alternatively you can usetype=substitutionas mentioned in thefontspecmanual (page 57) - I'm not sure if there is any difference.
– Marijn
7 hours ago
Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph
– Niranjan
7 hours ago
1
well if directlua is not defined that you didn't use lualatex (but probably xelatex).
– Ulrike Fischer
7 hours ago
@Niranjan if your question is about transliteration: with xelatex you could also use ateckitmapping. Simply search the site, you will find a number of answers.
– Ulrike Fischer
7 hours ago
Please see the edit.
– Niranjan
7 hours ago
add a comment |
with lualatex:
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,
usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]
begindocument
abcd
enddocument

with lualatex:
documentclassarticle
directlua
fonts.handlers.otf.addfeature
name = "shuffle",
type = "multiple",
data =
["a"] = "p",
["b"] = "r",
["c"] = "s",
["d"] = "t",
,
usepackagefontspec
setmainfonttexgyreheros%
[
RawFeature=+shuffle,
]
begindocument
abcd
enddocument

answered 8 hours ago
Ulrike FischerUlrike Fischer
207k9 gold badges314 silver badges708 bronze badges
207k9 gold badges314 silver badges708 bronze badges
Alternatively you can usetype=substitutionas mentioned in thefontspecmanual (page 57) - I'm not sure if there is any difference.
– Marijn
7 hours ago
Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph
– Niranjan
7 hours ago
1
well if directlua is not defined that you didn't use lualatex (but probably xelatex).
– Ulrike Fischer
7 hours ago
@Niranjan if your question is about transliteration: with xelatex you could also use ateckitmapping. Simply search the site, you will find a number of answers.
– Ulrike Fischer
7 hours ago
Please see the edit.
– Niranjan
7 hours ago
add a comment |
Alternatively you can usetype=substitutionas mentioned in thefontspecmanual (page 57) - I'm not sure if there is any difference.
– Marijn
7 hours ago
Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph
– Niranjan
7 hours ago
1
well if directlua is not defined that you didn't use lualatex (but probably xelatex).
– Ulrike Fischer
7 hours ago
@Niranjan if your question is about transliteration: with xelatex you could also use ateckitmapping. Simply search the site, you will find a number of answers.
– Ulrike Fischer
7 hours ago
Please see the edit.
– Niranjan
7 hours ago
Alternatively you can use
type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.– Marijn
7 hours ago
Alternatively you can use
type=substitution as mentioned in the fontspec manual (page 57) - I'm not sure if there is any difference.– Marijn
7 hours ago
Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph
– Niranjan
7 hours ago
Hello. Thanks for the help, but I am getting following errors. Undefined control sequence. directlua Missing begindocument. f fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. fontspec error: "font-not-found" The font "texgyreheros" cannot be found. Font TU/texgyreheros(0)/m/n/10=texgyreheros at 10.0pt not loadable: Metric (TFM) file or installed font not found. ] Overfull hbox (0.39941pt too wide) in paragraph
– Niranjan
7 hours ago
1
1
well if directlua is not defined that you didn't use lualatex (but probably xelatex).
– Ulrike Fischer
7 hours ago
well if directlua is not defined that you didn't use lualatex (but probably xelatex).
– Ulrike Fischer
7 hours ago
@Niranjan if your question is about transliteration: with xelatex you could also use a
teckit mapping. Simply search the site, you will find a number of answers.– Ulrike Fischer
7 hours ago
@Niranjan if your question is about transliteration: with xelatex you could also use a
teckit mapping. Simply search the site, you will find a number of answers.– Ulrike Fischer
7 hours ago
Please see the edit.
– Niranjan
7 hours ago
Please see the edit.
– Niranjan
7 hours ago
add a comment |
OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.
The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.
The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.
So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.
So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of
tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR
At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:
tokcycle[]<input stream>
expandedtokcycle[]<input stream>
begintokencycle[]
<input stream>
endtokencycle
beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle
The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).
Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.
Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).
Now for the code. First, the code to address the OP's problem:
documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
begindocument
%अबकड
verb|tcremap| handles a single token: tcremapअ.
texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.
noindenthrulefill
ENVIRONMENT, DIRECT APPROACH
begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
endtokencycle
noindenthrulefill
MACRO, TOKENIZED APPROACH
cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
toksrescancytoks
thecytoks
enddocument

And for the tokcycle code, which I hope to officially release in the next few weeks:
First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.
% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11
% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle
def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%
longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%
def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone
% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next
% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR
% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg
deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp
deftc@resetmacrolet@tcusemacrotc@empty
% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg
% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1
defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%
% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks
%--------------------------------
input listofitems.tex
% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12
% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%
% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi
catcode`@=12
endinput
Then there is the LaTeX style wrapper, tokcycle.sty:
ProvidesPackagetokcycle[V0.9]
usepackageenviron
%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR
% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR
% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR
% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR
input tokcycle.tex
endinput
Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.
Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.
– Sebastiano
3 hours ago
2
@Sebastiano I've been working on this about 5 weeks.
– Steven B. Segletes
3 hours ago
Impressive :-(. My best compliments.
– Sebastiano
3 hours ago
add a comment |
OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.
The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.
The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.
So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.
So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of
tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR
At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:
tokcycle[]<input stream>
expandedtokcycle[]<input stream>
begintokencycle[]
<input stream>
endtokencycle
beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle
The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).
Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.
Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).
Now for the code. First, the code to address the OP's problem:
documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
begindocument
%अबकड
verb|tcremap| handles a single token: tcremapअ.
texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.
noindenthrulefill
ENVIRONMENT, DIRECT APPROACH
begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
endtokencycle
noindenthrulefill
MACRO, TOKENIZED APPROACH
cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
toksrescancytoks
thecytoks
enddocument

And for the tokcycle code, which I hope to officially release in the next few weeks:
First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.
% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11
% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle
def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%
longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%
def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone
% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next
% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR
% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg
deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp
deftc@resetmacrolet@tcusemacrotc@empty
% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg
% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1
defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%
% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks
%--------------------------------
input listofitems.tex
% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12
% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%
% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi
catcode`@=12
endinput
Then there is the LaTeX style wrapper, tokcycle.sty:
ProvidesPackagetokcycle[V0.9]
usepackageenviron
%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR
% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR
% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR
% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR
input tokcycle.tex
endinput
Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.
Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.
– Sebastiano
3 hours ago
2
@Sebastiano I've been working on this about 5 weeks.
– Steven B. Segletes
3 hours ago
Impressive :-(. My best compliments.
– Sebastiano
3 hours ago
add a comment |
OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.
The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.
The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.
So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.
So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of
tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR
At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:
tokcycle[]<input stream>
expandedtokcycle[]<input stream>
begintokencycle[]
<input stream>
endtokencycle
beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle
The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).
Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.
Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).
Now for the code. First, the code to address the OP's problem:
documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
begindocument
%अबकड
verb|tcremap| handles a single token: tcremapअ.
texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.
noindenthrulefill
ENVIRONMENT, DIRECT APPROACH
begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
endtokencycle
noindenthrulefill
MACRO, TOKENIZED APPROACH
cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
toksrescancytoks
thecytoks
enddocument

And for the tokcycle code, which I hope to officially release in the next few weeks:
First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.
% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11
% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle
def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%
longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%
def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone
% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next
% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR
% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg
deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp
deftc@resetmacrolet@tcusemacrotc@empty
% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg
% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1
defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%
% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks
%--------------------------------
input listofitems.tex
% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12
% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%
% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi
catcode`@=12
endinput
Then there is the LaTeX style wrapper, tokcycle.sty:
ProvidesPackagetokcycle[V0.9]
usepackageenviron
%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR
% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR
% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR
% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR
input tokcycle.tex
endinput
Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.
OK, it's not quite ready for release yet, but I will use this opportunity to introduce the upcoming tokcycle package. It helps you to build tools to process tokens from an input stream. The idea here is that if you can build a macro to process an arbitrary single (non-macro, non-space) token, then tokcycle can provide a wrapper for processing an input stream on a token-by-token basis, using your provided macro.
The package approach is to categorize what comes next in the input stream as either a Character, Group, Macro, or Space. Your job in creating the token-cycle is to specify the LaTeX directives on how to handle each of those possibilities.
The package provides tools to help you build those directives. In addition, it supports what I call both the "direct approach" (direct typesetting) and the "tokenized approach" (build a token register). The direct approach is perhaps simpler, but has more limitations on its use, which don't apply to the tokenized approach.
So let us take the problem at hand. I need to build a macro that can take a single Character token input and provide a mapping to a different character (in a different font). Here is the code I propose for this:
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
It basically says, if I find a remap, use it, otherwise just output the original token. I provide a remap of 4 tokens as shown above.
So now let's get to the tokcycle syntax. It provides a Plain-TeX supported syntax (tokcycle.tex) of
tokcycraw
<end-of-group/cycle directive>
<Character directive>
<Group directive>
<Macro directive>
<Space directive>%
<token input stream>tcEOR
At the LaTeX level, it provides 2 macros and 2 environments, all of which call on the tokccycraw macro:
tokcycle[]<input stream>
expandedtokcycle[]<input stream>
begintokencycle[]
<input stream>
endtokencycle
beginexpandedtokencycle[]
<input stream>
endexpandedtokencycle
The LaTeX directives appear in the same order as tokcycraw, except that the first end-of-group/cycle directive is optional. The expanded versions apply expanded to the input stream before tokcycle processing (macros can be cordoned off with noexpand).
Quirks: if using the direct approach, cannot process input-stream macros that take an optional argument or more than one mandatory argument. If you plan to process macros that take a single argument, you have to employ the tcsavemacro/tcusemacro pair trick to hold back on executing the macro until right before the opening brace of its argument.
Different quirks for tokenized approach: Since bracing is added piecemeal to the token register, it has to be done in the form of catcode-12 braces (openbrace and closebrace), which are at the end, rescanned as grouping braces, using either scantokens or the provided toksrescan (this will actually save the rescan in the token register).
Now for the code. First, the code to address the OP's problem:
documentclassarticle
usepackageetoolbox,tokcycle,inputenc
deftcmapto#1#2expandafterdefcsname tcmapto#1endcsname#2
deftcremap#1ifcsdeftcmapto#1csname tcmapto#1endcsname#1
tcmapto अP
tcmapto बQ
tcmapto कR
tcmapto डS
begindocument
%अबकड
verb|tcremap| handles a single token: tcremapअ.
texttttokencycle and verb|tokcycle| handle a stream of
such tokens, including (within limits) embedded macros.
noindenthrulefill
ENVIRONMENT, DIRECT APPROACH
begintokencycle
[tcusemacro]
tcusemacrotcremap#1
tcusemacrogroupcycle
tcusemacrotcsavemacro
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
endtokencycle
noindenthrulefill
MACRO, TOKENIZED APPROACH
cytoks
tokcycle
Xaddcytokstcremap#1
xaddcytoksopenbracegroupcyclexaddcytoksclosebrace
addcytoks#1
addcytoks %
अबकड डड textitबकअ कड.
Other text for which no mapping is yet given as of today.
अबक done.
toksrescancytoks
thecytoks
enddocument

And for the tokcycle code, which I hope to officially release in the next few weeks:
First, the Plain-tex package tokcycle.tex. I've never written in plain-TeX before, so I hope I haven't screwed something up.
% tokcycle PLAIN-TeX FORM: tokcycraw...tcEOR
catcode`@=11
% ESSENTIAL METHOD: STREAMING MACRO WITH tcEOR TERMINATOR
% tokcycraw<end-group><chars><groups><macros><spaces>...tcEOR
deftc@empty
lettcEORtc@empty
def:lettc@sptoken= : % this makes tc@sptoken a space token
def@resetcytokscytoks
%
longdeftokcycraw#1#2#3#4#5%
@resetcytoks
tc@resetmacro
def@grpX#1%
def@chrT##1#2%
longdef@grpT##1#3%
longdef@macT##1#4%
def@spcT#5%
@tokcycle
def@tokcyclefuturelettc@next@@tokcycle
def@@tokcycle%
ifxtc@nexttcEORexpandafter@grpXexpandaftertc@resetmacroelse%
ifxtc@nexttc@sptokendeftokcycarg %
expandafterexpandafterexpandafter@@spcT%
else%
expandafterexpandafterexpandafter@@@tokcycle%
fi%
fi%
longdef@@@tokcycle#1%
deftokcycarg#1%
ifxbgrouptc@next
expandafter@@grpT
else
deftc@tmptc@ifmacro@@macT@@chrT%
expandaftertc@tmp
fi%
@tokcycle%
def@@chrTexpandafter@chrTexpandaftertokcycarg
def@@grpTexpandafter@grpTexpandaftertokcycarg
def@@macTexpandafter@macTexpandaftertokcycarg
% REMOVE ORIGINAL INPUT-STREAM SPACE AFTER PROCESSING SPACE DIRECTIVE
deftc@Space-B-Gone
def@@spcT@spcTexpandafter@tokcycletc@Space-B-Gone
% TO DETERMINE IF NON-GROUP (I.E., SINGLE) TOKEN IS CHAR OR MACRO
deftc@ifmacro#1#2%
expandafterifcatexpandafternoexpandtokcycargrelax
def@next#1elsedef@next#2fi@next
% DIRECTIVE FOR HANDLING GROUPS RECURSIVELY
defgroupcycleexpandafter@tokcycletokcycargtcEOR
% DIRECTIVES FOR DEFERRING THE USE OF MACROS IN THE "DIRECT APPROACH"
deftcsavemacro%
expandafterdefexpandafter@tcusemacroexpandaftertokcycarg
deftcusemacrolettc@tmp@tcusemacrotc@resetmacrotc@tmp
deftc@resetmacrolet@tcusemacrotc@empty
% DETOKENIZE CURRENT TOKEN/GROUP UNDER CONSIDERATION
defdetokcycargdetokenizeexpandaftertokcycarg
% FOR TOKENIZED APPROACH:
% THE newcytoks MACRO IS A PARAMETERIZED WAY TO ACCOMPLISH THE FOLLOWING:
% newtokscytoks
% newcommandaddcytoks[1]globalcytoksexpandafterthecytoks#1
% newcommandxaddcytoks[1]expandafteraddcytoksexpandafter#1
% newcommanddxaddcytoks#1expandafterxaddcytoksexpandafter#1
% newcommandXaddcytoks[1]xaddtoksexpanded#1
% GENERALIZED AS:
deftc@gobble#1
defnewcytoks#1%
edefcy@rootexpandaftertc@gobblestring#1%
ifdefined#1 ERROR:textbackslashcy@root already defined.
Overwriting.fi%
csname newtoksendcsname#1%
%
expandafterlongexpandafterdefcsname addcy@rootendcsname##1%
global#1expandafterthe#1##1%
%
def@tmpexpandafterdefcsname xaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname addcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname dxaddcy@rootendcsname####1%
expandafter@tmpexpandafterexpandafterexpandafter
csname xaddcy@rootendcsnameexpandafter##1%
%
def@tmpexpandafterdefcsname Xaddcy@rootendcsname####1%
expandafter@tmpexpandafter%
csname xaddcy@rootendcsnameexpanded##1%
% CREATE THE DEFAULT TOKCYCLE TOKEN REGISTER
newcytokscytoks
%--------------------------------
input listofitems.tex
% DEFINE CATCODE 12 BRACES
deftc@firstoftwo#1#2#1
deftc@secondoftwo#1#2#2
edeftc@bracepairdetokenize
edefopenbraceexpandaftertc@firstoftwotc@bracepair% _12
edefclosebraceexpandaftertc@secondoftwotc@bracepair% _12
% SET UP listofitems SEP CHARS FOR NESTED PARSING: _12/_12
deftc@sepchars%
expandafterexpandafterexpandaftersetsepchar
expandafterexpandafterexpandafter
expandafteropenbraceexpandafter/closebrace%
% CONVERT MATCHED CATCODE-12 BRACES IN TOKEN REGISTER TO GROUPING BRACES
deftoksrescan#1%
expandafterifxexpandafterrelaxthe#1relaxelse
bgroup
tc@sepchars
expandaftergreadlistexpandafter@parsetokexpandafterthe#1%
egroup
cytoks%
dxaddcytoks@parsetok[1]%
defredo0%
foreachitemzin@parsetok[]%
ifnumzcnt=1else
ifnumlistlen@parsetok[zcnt]>1relax
dxaddcytoksexpandafterexpandafterexpandafter
@parsetok[zcnt,1]%
foreachitemzzin@parsetok[zcnt]%
ifnumzzcnt=1else
ifnumzzcnt=2else
xaddcytoksclosebrace%
fi
xaddcytokszz%
fi
%
else
xaddcytoksopenbrace%
xaddcytoksz%
defredo1%
fi
fi
%
#1expandafterthecytoks%
ifnumredo=1relaxdef@nexttoksrescan#1elsedef@nextfi
expandafter@next
fi
catcode`@=12
endinput
Then there is the LaTeX style wrapper, tokcycle.sty:
ProvidesPackagetokcycle[V0.9]
usepackageenviron
%--------------------------------
% ARGUMENT FORM
newcommandtokcycle[6][]tokcycraw#1#2#3#4#5#6tcEOR
% expanded-ARGUMENT FORM
newcommandexpandedtokcycle[6][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpanded#6tcEOR
% ENVIRONMENT FORM
NewEnvirontokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksBODYtcEOR
% expanded-ENVIRONMENT FORM
NewEnvironexpandedtokencycle[5][]%
cytokstokcycraw#1#2#3#4#5%
expandaftertheexpandaftercytoksexpandedBODYtcEOR
input tokcycle.tex
endinput
Unfortunately, the documentation is too large to post here. But I would be happy to answer questions.
edited 3 hours ago
answered 4 hours ago
Steven B. SegletesSteven B. Segletes
167k9 gold badges210 silver badges429 bronze badges
167k9 gold badges210 silver badges429 bronze badges
Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.
– Sebastiano
3 hours ago
2
@Sebastiano I've been working on this about 5 weeks.
– Steven B. Segletes
3 hours ago
Impressive :-(. My best compliments.
– Sebastiano
3 hours ago
add a comment |
Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.
– Sebastiano
3 hours ago
2
@Sebastiano I've been working on this about 5 weeks.
– Steven B. Segletes
3 hours ago
Impressive :-(. My best compliments.
– Sebastiano
3 hours ago
Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.
– Sebastiano
3 hours ago
Excellent. I was wondering how you can give such detailed answers and how long it takes to do this. My best regards.
– Sebastiano
3 hours ago
2
2
@Sebastiano I've been working on this about 5 weeks.
– Steven B. Segletes
3 hours ago
@Sebastiano I've been working on this about 5 weeks.
– Steven B. Segletes
3 hours ago
Impressive :-(. My best compliments.
– Sebastiano
3 hours ago
Impressive :-(. My best compliments.
– Sebastiano
3 hours ago
add a comment |
Really, really bad idea :-)
You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.
I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:
documentclassarticle
newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup
begindocument
`abcd'remapchars`abcd'`abcd'
enddocument
It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?
– Niranjan
7 hours ago
@Niranjan Sorry, I have no idea about Devanagari. I did a test remappingदेp,वq,नाr,गs, andरीt, and wroteremapchars`देवनागरी'(test file) but it came out asp r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?
– Phelype Oleinik
7 hours ago
@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. Withremap दp remap वq remap नr remap गs remap रtandremapchars`दवनगर'it works. (I think I've probably just slaughtered the Devanagari script...)
– Phelype Oleinik
7 hours ago
Oh ok. Let me add the exact input and output in my question.
– Niranjan
7 hours ago
Please see the edit
– Niranjan
7 hours ago
|
show 1 more comment
Really, really bad idea :-)
You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.
I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:
documentclassarticle
newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup
begindocument
`abcd'remapchars`abcd'`abcd'
enddocument
It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?
– Niranjan
7 hours ago
@Niranjan Sorry, I have no idea about Devanagari. I did a test remappingदेp,वq,नाr,गs, andरीt, and wroteremapchars`देवनागरी'(test file) but it came out asp r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?
– Phelype Oleinik
7 hours ago
@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. Withremap दp remap वq remap नr remap गs remap रtandremapchars`दवनगर'it works. (I think I've probably just slaughtered the Devanagari script...)
– Phelype Oleinik
7 hours ago
Oh ok. Let me add the exact input and output in my question.
– Niranjan
7 hours ago
Please see the edit
– Niranjan
7 hours ago
|
show 1 more comment
Really, really bad idea :-)
You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.
I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:
documentclassarticle
newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup
begindocument
`abcd'remapchars`abcd'`abcd'
enddocument
Really, really bad idea :-)
You can make the characters you want to remap active and redefine them to print some other characters. However if you make, say, a active, then newcommand (for example) won't work anymore because TeX will understand it as newcomm and, where a gets replaced by a p.
I defined a command remap which takes to characters and makes the first one print the second one, and another remapchars command which should contain the remap instructions and then typesets the argument with the new settings and reverts them afterwards to avoid problems. Do not use remap “on the loose”. You've been warned. here you go:
documentclassarticle
newcommandremap[2]%
catcode`#1=active
begingroup
lccode`~=`#1%
lowercaseendgroupdef~char`#2ignorespaces
newcommandremapchars%
begingroup
remap ap
remap bq
remap cr
remap ds
innerremap
newcommandinnerremap[1]#1endgroup
begindocument
`abcd'remapchars`abcd'`abcd'
enddocument
answered 8 hours ago
Phelype OleinikPhelype Oleinik
30.9k7 gold badges52 silver badges105 bronze badges
30.9k7 gold badges52 silver badges105 bronze badges
It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?
– Niranjan
7 hours ago
@Niranjan Sorry, I have no idea about Devanagari. I did a test remappingदेp,वq,नाr,गs, andरीt, and wroteremapchars`देवनागरी'(test file) but it came out asp r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?
– Phelype Oleinik
7 hours ago
@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. Withremap दp remap वq remap नr remap गs remap रtandremapchars`दवनगर'it works. (I think I've probably just slaughtered the Devanagari script...)
– Phelype Oleinik
7 hours ago
Oh ok. Let me add the exact input and output in my question.
– Niranjan
7 hours ago
Please see the edit
– Niranjan
7 hours ago
|
show 1 more comment
It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?
– Niranjan
7 hours ago
@Niranjan Sorry, I have no idea about Devanagari. I did a test remappingदेp,वq,नाr,गs, andरीt, and wroteremapchars`देवनागरी'(test file) but it came out asp r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?
– Phelype Oleinik
7 hours ago
@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. Withremap दp remap वq remap नr remap गs remap रtandremapchars`दवनगर'it works. (I think I've probably just slaughtered the Devanagari script...)
– Phelype Oleinik
7 hours ago
Oh ok. Let me add the exact input and output in my question.
– Niranjan
7 hours ago
Please see the edit
– Niranjan
7 hours ago
It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?
– Niranjan
7 hours ago
It worked very nicely, this might be 'the' solution I am looking for, but it has not completely solved my problem yet. Right now I want to do this with an Indian script Devanagari. It uses inscript keyboard. I purposely omitted these details, because I found it unnecessary in the minimality principles. Unfortunately if I use your code to replace devanagari characters by roman, it does not work, what do you think is the reason?
– Niranjan
7 hours ago
@Niranjan Sorry, I have no idea about Devanagari. I did a test remapping
देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?– Phelype Oleinik
7 hours ago
@Niranjan Sorry, I have no idea about Devanagari. I did a test remapping
देp, वq, नाr, गs, and रीt, and wrote remapchars`देवनागरी' (test file) but it came out as p r t ‘q s ’. As I said, I have no idea about Devanagari, but to me this looks wrong. Can you post an example with the desired output, please?– Phelype Oleinik
7 hours ago
@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With
remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)– Phelype Oleinik
7 hours ago
@Niranjan Hm... Apparently some of those characters have combining diacritics. The code however only accepts single characters. With
remap दp remap वq remap नr remap गs remap रt and remapchars`दवनगर' it works. (I think I've probably just slaughtered the Devanagari script...)– Phelype Oleinik
7 hours ago
Oh ok. Let me add the exact input and output in my question.
– Niranjan
7 hours ago
Oh ok. Let me add the exact input and output in my question.
– Niranjan
7 hours ago
Please see the edit
– Niranjan
7 hours ago
Please see the edit
– Niranjan
7 hours ago
|
show 1 more comment
Thanks for contributing an answer to TeX - LaTeX Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2ftex.stackexchange.com%2fquestions%2f498243%2fauto-replacement-of-characters%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown

Are you open to using LuaLaTeX?
– Mico
8 hours ago
1
I use XeLaTeX very often, have never tried Lua, but yes, if it solves my problem, I'll use lua
– Niranjan
8 hours ago
1
You need to narrow down the application a bit, I think. For example, will this be argument or environment delimited? Will the transformation apply to groups within the main text? If so, it breaks arguments that specify things like lengths, where
2ptwould be converted into a nonsense unit. Likewise for color specification. If not applied to group content, then things liketextitabcwould not be converted. So try to tell us more about how this would actually be used.– Steven B. Segletes
8 hours ago
As @StevenB.Segletes commented: what is your use case exactly? Is it ok to use a command or environment, i.e.,
myreplaceabcdwhich is rendered aspqrs? Why do you want to do this wihtin LaTeX and not in your editor or with an external script? If you use Linux for example (or Mac or Cygwin in Windows) you can use thetrcommand in the terminal, orsedorawkorperletc. etc. to replace everything with a single command.– Marijn
8 hours ago
1
@Niranjan - Then you should be using Ulrike's answer.
– Mico
7 hours ago