     [file version.texi not included]

The micro-server Library
************************

This is the reference manual for the dmbcs-micro-server C++ library, version VERSION.

Copyright © 2018, 2020, 2025 DM Bespoke Computer Solutions

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

     A copy of the license is also available from the Free Software Foundation Web site at http://www.gnu.org/licenses/fdl.html.

Introduction
************

This document introduces, defines and exemplifies use of the dmbcs-micro-server C++ library, providing services to applications for processing and responding to requests from a web browser, including the possibility of making the application a stand-alone web server in its own right (a `micro-server').

HTTP Web Services
=================

A CGI program is a program, written in any language, which conforms to the Common Gateway Interface (CGI).  It is a program executed by a web server (on the server machine), and is passed information about the request from the user through the UNIX environment.  It is left to the program to interpret the user's requirements and to produce on its standard output channel a HTML document which the web server will relay back to the user's browser for display on his console.

CGI programs are thus very flexible and amenable to lots of jobs. Some of them are very small, amounting to only a couple of lines of an interpreted language to slurp a file from the local storage and send it straight back to the browser (which is essentially what a web server would normally do but this allows the programmer to do so in a more customized way).  On the other hand, some CGI programs are extremely large consisting of hundreds of thousands of lines of code which may interact with a database and other third-party systems (such as payment gateways) and do some real work on behalf of the user, before sending the results of the transactions along for the user's perusal.

This project came from a CGI program of the latter description.  Because of its size, complexity and the anticipated load (number of users), it was decided very early on that it would be written in a proper programming language which could be compiled optimized to run on the bare metal of the available computer system, whatever that is.  Thus it would have to be C or C++.  While the original implementation had been in C for political reasons, it has since been ported to the more convenient C++.  As a side-effect of this development, the parts of the system which deal specifically with the business of being a CGI program have been abstracted and encapsulated, and then they were syphoned off into their own separate library; this was further extended to provide the functions of a self-contained web server.  This is now being offered for general use.

The result is a library which can be used to inject a self-contained web server into an arbitrary C++ code base, opening up the interface to such a legacy program to the modern world, without needing to also build the infrastructure to serve such a program to the public (although in practice infrastructure needs to be built for other reasons---like security and virtualization---but the self-contained scheme allows for greater flexibility and better functional isolation).

Our Choice of Name
==================

The micro-server library was written from scratch since 2004 as part of a large internet project.  The requirement for a C++ framework for undertaking this task is very common, and thus it is no surprise that other efforts already exist, notably the GNU cgicc project.

Some consideration was given to the name of this project.  At first it was going to be called cgicc2, but that carries the implication that it somehow carries on from the GNU cgicc project, and that it is both a development and enhancement of it.  This is definitely not the case, but a parallel development, completely different but solving the same problems.  Unfortunately appending any number or even letter carries some implication of sequencing, and if the two projects continue (as they should) to be developed then some imbalance would occur in the rivalry.  Such is definitely not the intention, but for the two projects to be seen as separate.  Choice of one or other framework for any particular project should be based on what the framework offers, not what letters happen to come in its name.

Thus it is decided to mark this particular library with the string 'DMBCS': all code appears inside the namespace 'DMBCS::Micro_Server' and the include file is called 'dmbcs-micro-server.h'; the project is formally known as the 'dmbcs-micro-server' C++ library.

Your Choice of Library
======================

We offer only this advice: look at the API provided by all the available libraries, and consider the nature of the working framework (paradigm) that that implies; choose the library that seems most appropriate for your project: the one that you will feel most comfortable working with.

Outline of Library
==================

The library consists of a high-level wrapper and two workhorse classes. Http_Server interfaces with multiple clients and delegates processing of their requests to registered callback functions, which make direct use of the following workhorses.  Query_String looks after the part of the CGI interface which passes from the browser to the CGI program and provides methods specifically for interpreting the arguments from the browser's query string, and Hyper_Tags looks after the part of the CGI interface which passes back from the CGI program to the browser, and displays the results of the user's request; this deals with updating template HTML files with some dynamic content before passing the result to the web server for eventual transmission to the browser.

Using the library
*****************

To re-iterate: all code in this library comes in the namespace 'DMBCS::Micro_Server' and the header which introduces all of the functionality is 'dmbcs-micro-server.h'.

A project would generally consist of a template HTML file with markers where dynamically-generated content needs to be placed, a .css file which should be injected into the file or provided by the server as an inclusion target in the HTML, a .js file containing code that should run client-side to implement part of the functionality of the web site, and a (micro-) web server which runs continually, listens for client connections, and provides generated HTML files and JSON data strings on demand of the client.

Reference
*********

The utility functions
=====================

Introductory text here.

 -- Function: string slurp_file_with_error (const std::string &FILE_NAME) throw (Slurp_Error)

     This function will attempt to open the file named file_name (which may be a complete path to the file), and read the whole lot into a string. If any errors occur in the process, a Slurp_Error object will be thrown (these exception objects do not convey any information back to the application).

 -- Function: string slurp_file (const std::string &FILE_NAME)

     This function will attempt to open file FILE_NAME (which may be a full path), and will slurp the entire contents (which are presumably a hyper-tag'd template file) into a string.  If any errors occur, an empty string will be produced, and a message printed on the standard error channel to indicate the name of the file which has failed.

     In the majority of applications of this function, in the majority of calls, the error condition thrown from the function above will not happen; it would be regarded as an infrastructure build problem rather than a run-time error if a known file were not accessible.  Thus it is wasteful to worry about and deal with the error at global level--outside the library--, rather let the program carry on and provide a message to the user's console about the missing file (the fact that the problem occurred will be obvious!)  Once it is known which file has failed, the remedy will be equally obvious.

 -- Function: string htmlize_quotes (string && INPUT)

     Any occurrence of `` in INPUT will be replaced by `&quot;'

 -- Function: string htmlize (string && INPUT)

     All occurrences of HTML-special characters (<>&") will be replaced with entity representations.

     The main purpose of this method is to alleviate the possibility of injection attacks into a web-site; if any part of a page is generated from user-supplied input there is the danger that the user might be savvy enough to get JavaScript to execute there!  But the function is quite generally useful when it is known that a string needs to appear verbatim in a web page, regardless of its contents, and thus ought to be used whenever a string is added to a set of Hyper_Tags.

 -- Function: string strip_leading_space (string && INPUT)

     Replace all sequences of white space at the start of all lines with a single space character.

     HTML recognizes all sequences of white space outside of quotes as a single space separator.  Thus sequences of white space at the beginning of lines (which are used by web programmers to give a nice aesthetic indentation) are a waste of bandwidth.  This function will remove all such occurrences from INPUT.

 -- Function: string escape_quotes (string INPUT)

     Replace all occurrences of quotation marks and escapes with escaped versions, where the escape character is the back-slash.

     This function will put an escape ('\') in front of ''', '''', or '\' which occur in INPUT, and is useful (essential when dealing with user-supplied input) to ensure that strings do not mess up SQL statements.

 -- Function: string decode_url (string URL)

     This function will replace '+' with ' ', and any %-escaped character codes in the URL with the ASCII equivalent.

     This will produce a plain string, decoded from strings passed from forms or EcmaScript on web pages passed through parameter lists in URI's, as provided within the context of this library by the Query_String object.

 -- Function: string hexalize (unsigned char const *const BUFFER, size_t const BUFFER_SIZE)

     Return an ASCII string containing the hexadecimal representation of the octets in the given BUFFER.  The BUFFER_SIZE is the number of octets to decode.

 -- Function: string getenv (string const & ENV)

     Return the value of the environment variable called ENV, or an empty string if the variable does not exist in the environment.

     This is a convenience wrapper around the system getenv call, adapting it to the use of strings.

The Query_String class
======================

A Query_String object is a std::map<std::string, std::string> object in which the keys are the variables passed in the URL query string part and the values are the values from that query string.  Methods are defined which interrogate the class in details more specific to the CGI problem, and allow for values to be extracted in either string or a numeric form, as expected by the CGI program.

MACs
----

Constructors
------------

 -- Constructor: Query_String ()

     The null constructor, only useful if you are intending to build up a new query string for eventual inclusion into some part of a HTML page.

 -- Constructor: Query_String (const char *const *const ENV)

     The real workhorse constructor.  The argument ENV should be the environment variable passed to the CGI's main routine as the third argument.  This will create an object which represents the information passed to the CGI program via the query string, whether it were sent via the GET or the PUT protocols.  The object returned can subsequently be used to interrogate the user's request string.

Methods
-------

 -- Method: void Query_String::add (const std::string &NAME, const T &VALUE)

     This method will add an entry into the Query_String with key NAME and value a string representation of VALUE.  The type T of VALUE may be anything that can be passed into a std::ostream object via the << operator.

 -- Method: void Query_String::add (const std::string &NAME, const Query_String &INPUT, const T &FALLBACK)

     This method will add an entry into the current Query_String which is a copy of the entry in the INPUT Query_String, unless such entry does not exist in which case a key NAME with value a std::string representation of FALLBACK will be added to the current Query_String.  The type T of FALLBACK may be anything that can be passed to the << operator of a std::ostream object.

 -- Method: bool Query_String::posted () const

     Returns TRUE if the query string had been sent to the CGI program via the POST protocol, FALSE otherwise (which case would presumably have been caused by use of the GET protocol).

 -- Method: T Query_String::get (const std::string &name, const T &fallback) const

     This gets the value associated with the key NAME from the query string, unless no such key was present in which case FALLBACK is returned.  Note that the type of value returned is the same as that of FALLBACK, thus FALLBACK serves the dual role of defining the fallback value and determining the return type of the function.

 -- Method: std::vector <int> Query_String::get_list (const std::string &name) const

     In the case that a query string value corresponds to a multiple choice selection on the HTML form, it will be a comma-separated string of selected values.  This function will return a vector comprised of these values, interpreted as integers.  If there is no key NAME in the query string, then the returned vector will have no elements (zero length).

 -- Method: std::vector <int> Query_String::get_checked_array (const std::string &NAME) const

 -- Method: bool Query_String::has (const std::string &NAME) const

     This method returns TRUE if the NAME was a variable present in the query string, FALSE otherwise.

 -- Method: bool Query_String::mac_verified () const

     This method will return TRUE only if there was a MAC present in the query string, and the MAC was valid.  Otherwise FALSE will be returned.

Functions
---------

 -- Function: static string Query_String::add_mac (string INPUT)

     This static class member will add a cryptographic MAC onto any string, but this is only meaningful for composed query strings.

The Hyper_Tags class
====================

Whereas a HTML document contains markup tags expressed as labels between angle brackets (possibly in pairs), so a micro-server template file contains dynamic content holders expressed as labels between square brackets.  The Hyper_Tags class contains methods for replacing single tags with some dynamically-generated content, and for selectively keeping or removing sections of the template bracketed between pairs of hyper tags.

A Hyper_Tags object should be considered a container of tag definitions.  The CGI program will typically build up a list of such definitions in the container, until all the work is done when the template file is read in from local storage and the Hyper_Tags object is instructed to make all the changes in one go before sending the result to the server and hence to the user's browser.

Constructors_
-------------

 -- Constructor: Hyper_Tags ()

     The class only has the one, default, constructor, which returns an empty object waiting to be filled with tag definitions.

Methods_
--------

 -- Method: void Hyper_Tags::harden_last ()

     Normally, a later tag definition would overrule any earlier ones when substitutions are finally made into a template file.  However, if it is desired that the last tag added should be the final tag definition on substitution, then it can be `hardened' by calling this method.

 -- Method: bool Hyper_Tags::has (const std::string &NAME) const

     This method returns TRUE if a tag with the given NAME has been registered with the Hyper_Tags object.  In all other cases it returns FALSE.

 -- Method: Hyper_Tags &Hyper_Tags::add (const Hyper_Tags &REGISTRY, const int HARDEN = 0)

     This method copies all the tag definitions contained in REGISTRY into the current object, and returns a reference to the current object.  If the optional HARDEN argument is passed a non-zero value, then all the incumbent tag definitions will be hardened, as per the discussion of the harden_last () method above.

 -- Method: Hyper_Tags &Hyper_Tags::add (const std::string &NAME, string SUBSTITUTION_TEXT)

     One of the main methods of the class.  Declares that, in the final substitutions, any parts of the template file which look like ``[NAME/]'' should be replaced with SUBSTITUTION_TEXT.

     The return is a reference to the current object.

 -- Method: Hyper_Tags &Hyper_Tags::add (const std::string &NAME, const int value)

     Declares that tags called NAME in the template file be replaced with a string representing the integer VALUE.  The return is a reference to the current object.

 -- Method: Hyper_Tags &Hyper_Tags::add (const std::string &NAME, const double VALUE, const int PRECISION = 4)

     Declares that tags called NAME be replaced with a string representing the floating-point VALUE, with at least PRECISION significant figures.  The return is a reference to the current object.

 -- Method: Hyper_Tags &add (const std::string &NAME, Query_String &Q_S, const T &FALLBACK)

     Declares that tags called NAME be replaced with a value from Q_S with the same name if this exists, or with FALLBACK otherwise.  The type T of fallback should be one of std::string, int, or double.  The return is a reference to the current object.

 -- Method: void add_int_list (const std::string &NAME, I BEGIN, I END)

     Here, I is an iterator type over an STL container of int's.  This method declares that tags named NAME be substituted with a comma-separated list of integer values.

 -- Method: void add_date (const std::string &NAME, const time_t &TIME, const std::string &FORMAT)

     Specifies that tags named NAME be replaced with a string representing the date indicated by the UNIX TIME object.  The conversion takes place according to FORMAT, which is described in the strftime () libc manual.

 -- Method: void add_file (const std::string &NAME, const std::string &FILE_NAME)

     Specifies that tags named NAME in the template HTML file be replaced with the entire contents of the file called FILE_NAME in the file system.

Method void add_passthrough_tag (const Query_String &QUERY_STRING, ...)

The trailing arguments are a list of const char *const, terminated with a NULL pointer.  This method specifies that contents of the HTML template file which look like ``[pass-through/]'' be replaced with a set of lines which look like ``<input type=``hidden'' name=``name'' value=``value''/>'' where the NAMES are taken from the list of arguments to the function, and the __VALUES__ are taken from the QUERY_STRING variables with the same names.  Thus, if the tag in the template file is contained within ``<form>'' markup elements, the current values in the query string will be propagated into any query string that the new form produces.

Type enum Hyper_Tags::Section_Keep { KEEP_SECTION, LOSE_SECTION }

Method Hyper_Tags &section (const std::string &NAME, const Section_Keep &KEEP)

This method declares the action to be taken when a pair of tags like ``[NAME]'' and ``[/NAME]'' are seen in a template file. If KEEP is KEEP_SECTION, then the tags are simply removed from the file leaving the contents between the tags intact.  If KEEP is LOSE_SECTION then the tags and all the text between them is removed from the file.

Method string Hyper_Tags::substitute (string HTML_TEMPLATE, int REPEAT = 1)

One of the main methods of the class.  This will go through HTML_TEMPLATE and replace any hyper tags which have been registered with the substitution text that has been declared as the replacement.  If a tag is seen in the HTML_TEMPLATE which has not been registered with the current object, then this tag will remain intact in the template.

Additionally, any pairs of tags seen in HTML_TEMPLATE which have been registered as a section will be removed, possibly removing also all the text between the tags, according to the registered behaviour.

Functions_
----------

 -- Function: static string Hyper_Tags::strip_loose_tags (string INPUT)

     It may happen that, after processing, a HTML template string still contains some hyper tags which have not been substituted.  This static function will injudiciously remove any such tags, thus eliminating any ugliness from the page sent back to the user.

The Http_Server class
=====================

Constructor__
-------------

 -- Function: Http_Server::Http_Server (const int SOCKET, const
     std::map<std::string, std::function<void (Query_String, int)> METHODS)

     Create a HTTP server to listen on the SOCKET.  Processing for requests will be delegated to the METHODS, referenced by the HTTP-call path name via the map.  When called, the METHODS will be passed a Query_String object and a socket on which a response must be sent; methods are provided in the Http_Server namespace to facilitate with this.

     Please see the example application to see how this is used in practice.

Functions__
-----------

 -- Function: bool tick (Http_Server &SERVER, const long WAIT_US)

     Cause the SERVER to act on a single instruction from a web client, waiting up to WAIT_US micro-seconds for such a request to appear.

     The return is usually TRUE, but may be FALSE if the listening socket goes away (this is a mechanism that could be used by a request processing function to terminate the server frow within).

     In practice this method is called repeatedly, ad infinitum.

 -- Function: void Http_Server::return_html (const int SOCKET,
     const std::string &HTML)

     This method sends the HTML to the socket, dressed up as a valid response to a HTTP request for data.

 -- Function: void Http_Server::return_text (const int SOCKET,
     const std::string &TEXT)

     This method sends the TEXT to the socket, dressed up as a valid response to a HTTP request for data.

 -- Function: void Http_Server::return_pdf (const int SOCKET,
     const std::string &PDF)

     This method sends the PDF to the socket, dressed up as a valid response to a HTTP request for data.  The `string' is actually understood as a buffer of arbitrary octects here.

 -- Function: void Http_Server::return_json (const int SOCKET,
     const std::string &JSON)

     This method sends the JSON to the socket, dressed up as a valid response to a HTTP request for data.

 -- Function: void Http_Server::return_bad (const int SOCKET)

     This method sends an empty payload to the socket, dressed up as a valid response to a HTTP request for data (it would generally be taken by an Ecmascript application to be a `bad' response to a request for data).

 -- Function: void Http_Server::return_png (const int SOCKET,
     const std::string &PNG)

     This method sends the PNG to the socket, dressed up as a valid response to a HTTP request for data.  The `string' is actually understood as a buffer of arbitrary octects here.

 -- Function: void Http_Server::return_svg (const int SOCKET,
     const std::string &SVG)

     This method sends the SVG to the socket, dressed up as a valid response to a HTTP request for data.  The `string' is actually understood as a buffer of arbitrary octects here.

Examples of Use
***************

We don't start with the tradition hello, world program as this would not demonstrate in any way the dynamic nature of CGI programming!  The simplest example we can think of allows the user to input two numbers and returns the product of them, and the full code for this is provided in the next section.

Simple example -- Online multiplier
===================================

The following is provided as the simplest code to demonstrate use of the dmbcs-micro-server library, not to inform of any coding style or quality system approach.  We assume a standard GNU system with recent make, bash, gcc, etc.

Start with the HTML file calc.html

    <html>
      <head><title>Multiplier</title></head>
      <body><h1>Multiplier</h1>
        <form action="compute" method="GET" id="calc">
          <input type="text" name="arg_1"/> x <input type="text" name="arg_2"/>
                = <input type="text" id="result" value="[result/]"></input>
          <br>
          <input type="submit">CALCULATE</input>
        </form>
      </body>
    </html>

And the C++ source file calc.cc

        #include <dmbcs-micro-server.h>

        using namespace DMBCS::Micro_Server;

        /*  This function both serves up the basic HTML page, and
         *  performs the multiplication and injects the result into the
         *  HTML. */
        void  home_page  (Query_String const &query, int const socket)
        {
            Hyper_Tags  tags;

            tags.add  ("result",
                       query.get ("arg_1", 0) * query.get ("arg_2", 0));

            Http_Server::return_html (socket,
                                      substitute (tags,
                                                  slurp_file ("calc.html")));
        }

        int main ()
        {
           auto server  =  Http_Server {2022,
                                        { {"", home_page},
                                          {"compute", home_page} }};

           for (;;)   tick  (server, 1000000);

           return 0;
        }

then the makefile

        CXXFLAGS = `pkg-config --cflags dmbcs-micro-server`
        LDFLAGS  = `pkg-config  --libs  dmbcs-micro-server`

Then at the command line type

        make calc
        ./calc

then point a browser at http://localhost:2022/ and use the simple calculator (don't try to do anything funny: the code has been kept deliberately simple and doesn't do any error checking).  Note that an answer can be obtained directly with a URL like http://localhost:2022/compute?arg_1=3&arg_2=4.

Hints on how the library might be used in a real application
************************************************************

While the basic library as it is serves perfectly well for small CGI programs, the library was derived from a larger project.  From this experience, it is seen that many large projects will want to produce specializations of the classes and utility functions provided by the library for the particular project.  The following are some specific details about how the library was enhanced in the original project.

Web-site skins
==============

The underlying paradigm promoted by the library is one where the programmer supplies template HTML files which the CGI program populates with extra dynamic content.  This opens the possibility of supplying the same dynamic content into different templates, for example where a web site allows the user to choose a favorite `skin', or `look-and-feel', then each template file would exist in different forms to provide all the different skins available.  In this case, the library can be extended by ensuring that whenever a template is slurped from local storage for use, it is the template appropriate for the user's choice of skin that is pulled up.

In the original project, this was achieved by placing all templates for a skin into a single directory to which the CGI program has access (it was actually a sub-directory underneath the cgi-bin directory, and the web server was configured to *not* allow direct access from a browser to these files).

The main modification was to add a new utility function in the project's namespace

string slurp_file (const std::string &skin, const std::string &filename)

which is called by the program in lieu of slurp_file, and ensures that the file from the correct skin directory is chosen, calling through to the library function to get the eventual work done.  To complete the effect, it is also necessary to derive a new class from Hyper_Tags with the following additional methods

blah, blah, blah

(The Query_String class works completely as is; it may be convenient to introduce this into the projects own namespace with a

typedef Query_String Query_String;

line in a project header file.)

Other inheritence
=================