\documentclass[12pt,letterpaper,openany]{article}
\title{pua2ot: A Utility for Junicode}
\author{Peter S. Baker}
\date{\today}

\usepackage[MainFeatures={Language=English,StylisticSet=9}]{junicodevf}
\setsansfont{Noto Sans}[Scale=MatchLowercase,Numbers=Lowercase]

\usepackage[hypertexnames=true, plainpages=false, pdfpagelabels]{hyperref}
\hypersetup{pdftex, colorlinks=true, linkcolor=blue, citecolor=blue, filecolor=blue,%
  urlcolor=blue, pdftitle=, pdfauthor=, pdfsubject=, pdfkeywords=,}

\usepackage{graphicx}
\usepackage[table,dvipsnames]{xcolor}

\definecolor{GGOrange}{RGB}{240,74,6}
\definecolor{BrickRed}{RGB}{146,18,6}
\definecolor{myLightPurple}{RGB}{138,43,226}
\newcommand\textSourceText[1]{{\color{GGOrange}\textsf{#1}}}
\newcommand\textex[1]{\textrm{\textbf{\color{BrickRed}#1}}}
\newcommand{\unic}[1]{{\jSmCond\addfontfeature{Numbers={Uppercase,Monospaced}}#1}}
\newcommand{\pto}{\textsf{\color{myLightPurple}pua2ot}}

\begin{document}

\maketitle

\section{The scripts}

Modern best practice, when typesetting documents or creating web pages, is to
avoid the use of characters from the Unicode Private Use Area (PUA) whenever
possible, instead preferring standard Unicode characters styled with OpenType
features.\footnote{For an explanation, see the \textit{Junicode Manual}, §\,4.1} 
The Medieval Unicode Font Initiative (MUFI), founded before OpenType
was widely supported by software, defines more than 850 PUA characters, and
Junicode includes all of them. But the font also offers more-accessible 
alternatives to all but a handful of these characters.

OpenType is a powerful technology, but it is undeniably more difficult 
to work with than MUFI’s PUA characters and entities. To copy any character (or its 
corresponding entity) from the MUFI website and paste it into your text is a 
straightforward procedure. To do the same job with OpenType features is more 
complex, requiring users to locate the correct Unicode characters
and features and to figure out how to make them work together in various software 
programs.

{\pto} automates this task. For each PUA character or MUFI entity in a 
text or HTML document, the script substitutes a \textbf{base} consisting of 
one or more standard Unicode characters and, if necessary, applies one or 
more \textbf{methods} (grounded in OpenType technology) to style this base. For 
many characters, more than one base is available, and more than one method.

The methods employed by {\pto} are these:

\begin{description}
  \item[otag (OpenType tag).] One or more OpenType features is turned on for a 
  character or sequence of characters in much the same way that, say, italics
  can be applied to a run of text in a word processor: it is a modal method,
  which has the advantage of being legible and widely understood but can greatly
  swell the size of a text containing many PUA characters.

  \item[utag (Unicode tag).] An alternative to the otag method is one (exclusive
  to Junicode) that uses characters from the Unicode “Tags” range (U+E0020–E007E)
  as a modeless style of markup in which a single base character is modified by 
  two following tag characters. When used in a browser, tag characters are 
  invisible and do not interfere with searches, but their invisibility can make 
  them difficult to work with. {\pto} can use entities for tag characters to 
  make them visible: the entities are resolved to tag characters when the text 
  is displayed.

  \item[zwj (Zero-width joiner).] The zwj character (U+200D) can be used to signal 
  that two characters should be joined in a ligature: it provides an alternative 
  to otag-style markup. For example, the PUA code point U+EBAC is a ligature of 
  long \textex{s} (ſ) and insular \textex{v} (ꝩ): to obtain an equivalent one can 
  apply the OpenType feature \textSourceText{hlig} (Historic Ligatures) to the 
  base \textex{ſꝩ}—or one can place a zwj between the two characters. This method 
  is similar to the utag method in that it is modeless and employs a character
  that is typically invisible.

  \item[entity (Junicode entities).] For technical reasons, combining marks 
  encoded in the PUA should not be used, since software cannot position them 
  correctly over their bases. One alternative is to use Junicode entities,
  which resemble HTML entities but begin with an underscore. For example, in the
  sequence \textSourceText{a\&\_upmc;}, the entity \textSourceText{\&\_upmc;}
  represents an upward-curving macron, which will be correctly positioned
  over the preceding a: \textex{\addfontfeature{StylisticSet=10}a\&\_upmc;}.

  \item[enla (Enlarge axis).] For some forty-five of MUFI’s “enlarged” letters,
  used to represent the \textit{litterae notabiliores} of medieval manuscripts,
  one can use Junicode’s Enlarge axis, which allows one to control the size 
  of the letters precisely (on a scale from 0 to 100, where 0 produces a 
  letter the size of other lowercase letters, and 100 produces a lowercase 
  letter the size of a capital—the default value is 32) without using either 
  MUFI’s PUA characters or Junicode’s \textSourceText{ss06} “Enlarged minusucles”
  feature.
\end{description}

Alternate bases are grouped as follows (their names are followed by the 
identifiers to be used with the “Alternate bases” option, discussed below):

\begin{description}
  \item[Insular (\textSourceText{insular}).] By default, {\pto} replaces Unicode 
  characters described as 
  “insular” because these characters are often not searchable as their 
  alphabetic equivalents. However, since some projects have have good reasons 
  to retain these characters, {\pto}'s default behavior can be overridden (see 
  “Options” below for details).

  \item[Small caps (\textSourceText{smallcap}).] MUFI PUA characters described as 
  “small capitals” are 
  by default replaced by normal alphabetic characters with OpenType markup applied 
  (typically \textSourceText{pcap} “Petite capitals”). But Unicode has characters 
  labeled “small capital.” These are not intended as variants of alphabetic
  characters (they are instead phonetic characters), and in some software they 
  can’t be searched by their apparent alphabetic equivalents. But if you 
  prefer, the Unicode small caps can be used as a base.

  \item[Punctuation (\textSourceText{punctuation}).] MUFI encodes a number of 
  punctuation marks in the PUA.
  Often it is difficult to find a plausible Unicode base for these marks;
  instead, Junicode uses the period as a base for all of them. Where
  plausible bases are available, {\pto} uses them by default; but you can
  use the period as a base if you prefer.

  \item[Marks (\textSourceText{mark}).] As with punctuation, so with marks. 
  All of MUFI's combining 
  marks can be handled in Junicode as variants of the macron (U+0304), but 
  where better bases are available, they are used by default. If you prefer,
  you can use the macron for all PUA-encoded punctuation marks.

  \item[Currency (\textSourceText{currency}).] All of MUFI's PUA-encoded currency
  symbols are treated by 
  Junicode as variants of U+00A4, Unicode's generic currency sign, but where 
  plausible Unicode bases are available, those are used instead. If you prefer,
  it is possible to use U+00A4 as the base for all PUA-encoded currency symbols.

  \item[Miscellaneous alphabetic (\textSourceText{alpha}).] For a few alphabetic 
  characters (U+E7E4 , 
  U+EEFD , U+EEFE , U+F0A7 , U+F0A8 , U+F127 \,) more than one base can be 
  used. The defaults here are chosen to promote accessibility, but these 
  defaults can be overridden.
\end{description}

\noindent For both methods and bases, {\pto} consults a priority list: methods or base 
classes are listed in order of preference: if more than one method or base class 
is listed for a PUA character, the one that comes earliest in the priority list 
is preferred. These lists are discussed further in the “Options” section below.

Before you do anything else with {\pto}, copy the two font files
\textSourceText{JunicodeVF-Roman.woff2} and \textSourceText{JunicodeVF-Italic.woff2}
into the \textSourceText{pua2ot/fonts} directory in the unzipped
Junicode file tree.

\subsection{With Graphical User Interface}

To use {\pto} with a GUI, find the file \textSourceText{pua2ot-gui.html} in
the \textSourceText{pua2ot/gui} directory and double-click or
drag it into an up-to-date web browser. The program will look like this:

\begin{figure}[h!]
  \centering\includegraphics[width=\textwidth]{pua2ot.jpg}
\end{figure}

\noindent The large text-box on the left (initially displaying an excerpt from the 
Middle English \textit{Ancrene Wisse}) is the \textbf{Source Box}; it shows the 
text with PUA characters or entities. The text-box 
on the right is the \textbf{Destination Box}: it displays the result of the 
conversion. These two boxes should look the same: their sameness assures you that 
the conversion has gone well.

Immediately below the text boxes are options that affect their operation. 
To load a text or html file, click the “Choose File” button (“Browse” in some 
browsers) or drag the file into the 
Source Box. To edit the text, check the “Edit Source” box: you can type or 
paste in text when editing is enabled. To see which
characters in the source are being replaced, check the “Mark Replaced Chars in 
Source” box. To view the code that underlies the text displayed in the
Destination Box, check the “Show Code” box. To export the converted text,
click the “Download” or the “Copy” button.

To skip the conversion of any element of an HTML file, add the class
\textSourceText{noconv} to the element. You can add a 
\textSourceText{{\textless}span{\textgreater}} with 
class \textSourceText{noconv} to skip an arbitrary stretch of text.

The other options on the page affect the conversion. These are explained below,
in the “Options” section.

\subsection{Embedded in an HTML file}

The second version of the script converts the {\textless}body{\textgreater}
element of any HTML page before it is displayed, with the result that characters 
that could not be searched will become searchable, and the accessibility of the
page will be improved. If the HTML file contains static text, make sure the 
script file, \textSourceText{pua2ot-embed-min.js}, is
in the same folder as your HTML file. Then include 
these lines at the very end of the {\textless}body{\textgreater}:

\begin{scriptsize}\begin{verbatim}<script src="pua2ot-embed-min.js"></script>
<script class="noconv">
    // options.utagEntities = false;
    // options.utagLookup = UTAG_DICT;
    // options.nonWordTags = ["ss04", "ss05", "ss06", "pcap", "smcp", "hlig"];
    // options.keepUnicode = false;
    // options.methodPriority = "compact";
    // options.prefList = ["utag", "zwj", "otag", "entity"];
    // options.defaultTags = {"ss10": 1, "cv69": 7};
    // options.language = "en";
    // options.basePreferences = [];
    // options.enlargedScale = 32;
    // options.charSkip = [];
    convertAll();
</script>\end{verbatim}\end{scriptsize}

\noindent If your web page loads text dynamically, it won’t work 
simply to include the command \textSourceText{convertAll()} at the end of the 
HTML file, as it will likely be executed before the text is loaded. Instead,
have the process that loads the text trigger an event when the loading is done 
and call \textSourceText{convertAll()} from inside the event handler.

In the html file, include \textSourceText{noconv} in the class list of any element whose
content you don't want converted: the script will skip over such elements.

In the example above, the commented lines before the \textSourceText{convertAll()} 
command are options, explained in a later section.

\subsection{Node module}

The file \textSourceText{pua2ot-common.js} includes instructions for using it as
a Node module, but if you are not making changes in the programming you can use
\textSourceText{node/pua2ot-node-min.js}, which packages the {\pto}
database and program code in a single minified file. The file exports the object
\textSourceText{options} (the options it supplies are listed in the next section)
and two functions: \textSourceText{pua2otNode.cleanup\_string(s)}, which collapses
all sequences of whitespace characters into single spaces, and 
\textSourceText{pua2otNode.convert(text\_buffer, repl\_ent)}, which converts and returns
the text \textSourceText{text\_buffer}, first replacing any MUFI entities if
\textSourceText{repl\_ent} is true.

For an example of how the Node module can be used, consult and run the file 
\textSourceText{test2.js}. Note especially the \textSourceText{require} statement in 
line 3, the \textSourceText{options} object in line 13, and the invocation of 
\textSourceText{convert()} in line 15.


\subsection{Reference page}

In the \textSourceText{src} folder is a file \textSourceText{charlist.html}. Drag this 
into a web browser to view a list of all the transformations that {\pto} is capable
of. Originally intended as a tool for proofing the database that powers {\pto},
the page can also be used to look up the methods for individual PUA codes. Type the
hexadecimal code for any of MUFI's PUA characters into the “Filter” box. Alternatively,
type any word or phrase (e.g. “combining,” “small capital”) to search for in the 
“description” column. To view the hexadecimal codes for the Unicode base for any 
conversion, click on a cell in the “base” column. To view the underlying code for 
any character, click on any of the last five columns of the row for the character 
you are interested in. You can copy any of these methods onto the clipboard and 
paste them into a document: just click the “Copy” button in the dialog that appears
when you click on one of the last five cells in a row.


\section{Options}

In addition to the checkboxes, menus, and text-entry blanks in the web app, the
JavaScript program has an \textSourceText{options} object whose properties can be
manipulated to affect the program's output.

\subsection{More legible markup}

Program default: \textSourceText{options.methodPriority="compact"}.\\
By default, {\pto} uses the \textbf{utag} method for most of its 
conversions of single characters and the \textbf{zwj} method for most ligatures: these 
two methods occur first in the “compact” preference list. Because the Unicode tags and
zero-width joiner employed by these methods are zero-width and invisible, they can
make it difficult to diagnose problems in your code. If you set this option to 
“legible,” {\pto} will prefer the \textbf{otag} and (where appropriate) \textbf{enla} 
methods, which insert HTML tags into the text so that you can see clearly what the code 
is doing. If you choose the “legible” option to generate a web page, the HTML file 
for the page will be noticeably larger than with the “compact” option: you should 
consider running the output of this script through a utility (e.g. the 
\href{https://www.npmjs.com/package/html-style-converter}{Javascript HTML Style 
Converter}) that converts inline styles to a CSS stylesheet.

\subsection{options.prefList}

Program default: \textSourceText{options.prefList = [\char"0022utag\char"0022, \char"0022zwj\char"0022,
\char"0022enla\char"0022, \char"0022otag\char"0022, \char"0022entity\char"0022]}.\\
Not available in the GUI flavor. Use this option to install a custom priority 
list instead of the two predefined lists labeled “compact” and “legible.”
Methods that come earlier in the array are preferred over those that come later.

\subsection{Use entities for Unicode tags}

Program default: \textSourceText{options.utagEntities=false}.\\
If \textSourceText{true}, {\pto} uses entities to represent Unicode tags. (These are not 
HTML entities, but unique to Junicode.) This makes the code produced by this script more 
legible, but the entities make the page somewhat larger and can interfere with searching.
The best use of this feature may be to help you spot problems with your Unicode tags
and turn it off before publishing your text.

\subsection{Keep problematic Unicodes}

Program default: \textSourceText{options.keepUnicode=false}.\\
“Problematic Unicodes” 
are characters that are standard Unicode (not PUA) and look like 
variants of common characters, but that many applications do not recognize as 
variants. Examples include dotless i (\textex{ı}, \unic{U+0131}) and the “insular” 
letter-shapes (e.g. \textex{ꝺ}, \unic{U+A77A}). By default, this script replaces these 
characters with ones that can be searched as letters of the alphabet (e.g. \textex{ꝺ} as
\textex{d}). To retain the “problematic Unicodes,” likely simplifying your markup and
making your file smaller, but making your text less accessible in some applications, 
check this box in the GUI version or set \textSourceText{options.keepUnicode=true}.

\subsection{Alternate bases}

Program example: \textSourceText{options.basePreferences=["insular", "punctuation"]}.\\
As a large font with many OpenType features, Junicode often provides more than one way 
to achieve the outcome you want. This option affects the character classes
mentioned above, pp. 2–3. Select one or more of the items in the GUI list (use the 
Shift or Control/Command keys to make multiple selections) or assign an array
containing the identifiers for these classes to \textSourceText{options.basePreferences}:
{\pto} will use alternate bases for the classes you select.

\subsection{options.replaceMUFIEntities}

Program default: \textSourceText{options.replaceMUFIEntities=true}.\\
If \textSourceText{true}, MUFI’s entities (those that
resolve to PUA code points) are replaced by those code points before any other processing
takes place. Set this to \textSourceText{false} if the text you want to convert contains 
no entities and you want to save a little time processing a large text.
Not available in the GUI, where entities are always converted.

\subsection{Default features}

Program default: \textSourceText{options.defaultTags=\{\char"0022ss10\char"0022:\,1,\,\char"0022cv69\char"0022:\,7\}}.\\
GUI default: \textSourceText{ss10 cv69[7]}.\\
You may wish some OpenType features to be on for the whole document. For example, 
\textSourceText{ss10} “Entities and Tags” must be on in any document that uses Junicode entities or 
Unicode tags. To manage 
your text’s OpenType features, this script needs to know about these default features; 
and when you download or copy your text, they will be recorded in an enclosing {\textless}div{\textgreater}. 
Note the formats: In the GUI this is a space-separated list of four-letter OpenType tags; if you need 
to add an index, do so in square brackets immediately after the tag (the format used in the
\textit{Junicode Manual}). Elsewhere it is a comma-separated list of key-value pairs, where the keys 
(four-letter tags identifying OpenType features) are in double quotation marks, followed 
by a colon and a number (1 means “on,” but some features (especially the \textSourceText{cvnn} 
“Character Variant” features) require a numerical index).

\subsection{Non-word tags}

Program default: \textSourceText{options.nonWordTags=[\char"0022ss04\char"0022,\char"0022ss05\char"0022,\char"0022%
ss06\char"0022,\char"0022pcap\char"0022,\char"0022smcp\char"0022,\char"0022hlig\char"0022,%
\char"0022enla\char"0022]}.\\
GUI default: \textSourceText{ss04 ss05 ss06 pcap smcp hlig enla}.\\
When “More legible markup” 
is checked, the script applies most <span> tags to whole words, but 
a few to individual characters. This distinction is based on an educated guess as to which OpenType 
features are appropriate to apply to whole words. Sometimes this guess will be wrong. For example, the 
word \textit{rectificaꞇionem} (about halfway through the sample \textit{Ancrene Wisse} text) contains two different 
styles of \textex{t}, but when \textSourceText{cv40} (which transforms default \textex{t} into insular 
\textex{ꞇ}) is applied to the whole word, 
that difference is erased. If you add \textSourceText{cv40} to this list, the feature \textSourceText{cv40} 
will always be applied to single letters rather than whole words. In the GUI version of {\pto},
this is a space-separated list of four-letter tags (only tags used by this program are accepted here);
elsewhere these tags must be enclosed in 
quotation marks and separated by commas, and the whole list must be enclosed in square brackets.

\subsection{Language}

Program default: \textSourceText{options.language=\char"0022en\char"0022}.\\
It is important to set the language properly, both here and in your document, since this script
and some of Junicode’s OpenType features behave differently depending on the language. If your 
language is not in the drop-down list (available
only in the GUI), choose “Other” and type in the two-letter code for the language (you can look
it up \href{https://www.w3schools.com/tags/ref_language_codes.asp}{here}.).

\subsection{Enlarge axis scale}

Program default: \textSourceText{options.enlargedScale=32}.\\
When \textSourceText{options.methodPriority="legible"}, or when you use a custom priority list 
with \textSourceText{“enla”} near the top, {\pto} will invoke Junicode’s Enlarge Axis for 
“enlarged minuscules.” One advantage of this method is that you can precisely control their 
size on a scale from 0 (the letter is the same size as other lowercase letters) to 100 (the
letter is the same size and weight as a capital). The default is 32, which makes it the same 
size as the enlarged minuscules produced by other means.

\subsection{Skip character}

Program example: \textSourceText{options.charSkip=["0131", "A787", "A77A", "F0AE"]}.\\
GUI example: \textSourceText{\jRegular ı A787 A77A }.\\
To cause {\pto} to skip over all instances 
of a character, enter it here. In the GUI, you can type or paste a space-separated 
list of characters and hexadecimal codes in the blank. In programming, only hexadecimal 
codes are allowed; they must be
enclosed in quotation marks and separated by commas, and the whole list must be enclosed in 
square brackets.

\end{document}