FreeM Developer Blog

It may or may not be known to most readers that FreeM has been involved in an on-again, off-again relationship with the current incarnation of the M Development Committee, the organization that presently owns the intellectual properties involved in previous M standards, such as those released in 1977, 1984, 1990, and 1995, variously under the aegis of ANSI and/or ISO.

This organization has been promising a new standard on which the M developer, implementer, and user community could hang its hat as we move into the future. The timing of these efforts is, in our opinion, critical: M faces routine smears to the effect that it is a stagnant, legacy language, from which responsible enterprise IT organizations should have a well-defined migration path, and which suffers from a lack of features necessary for the development of modern and future-proof applications. A new and bold standard, along with implementers willing to take risks in creating new markets for the language, is, in our opinion, of vital importance.

It is our considered opinion that the path forward for M in its role as not just a high-throughput “NoSQL” database, but as a language which couples its highly-capable database with a powerful and expressive language, is to make bold moves—even those which may break direct compatibility with its legacy implementations—in both its official standards and its implementations.

M's community is small but boasts of pockets of deeply dedicated engineers whose passion for the language and the unique power of its tight coupling to a powerful database cannot be overstated. However, its standards organization is, in our opinion, selling it short in many critical ways that we will uncover and explore further here:

First, MDC leadership consists of appointed, non-elected members of a well-known M consulting organization. In addition to the more obvious potential for conflicts of interest, the leadership has routinely conducted large portions of its work in subterfuge, producing decisions by executive fiat and with no community input that set critical aspects of the organization's posture and direction from an ivory tower to which the community has very limited and curated access. It has been routinely promised that a future phase of standards development would open up the organization to community inputs, but given the inertia of the current phase and the critical decisions taken during its execution, the ability of the community to redirect the organization in an agile and democratic fashion is deeply curtailed. Statements like “the next standard will do X” have routinely been made, indicating that elements of the next standard have been predetermined, immune to any community feedback.

Second, it has been stated that the organization would have an executive board appointed by the current leadership, prior to opening the organization up to community input. Though this proposed board has been promised to be one that simply keeps the organization on a productive track, it is well-known to most adults that such broad powers, once given, are rarely returned, and the potential for corruption is vast. Also, the definition for what the correct track is was made without community input. Such a small and tight-knit community deserves far greater influence over the formation of a standard and governance of a standards organization to which it will be beholden, especially in the formative stages of the organization's reconstitution.

Third, this proposed new standard has been promised and under intermittent development for at least the past twelve years. When the tight and closed nucleus of its leadership becomes distracted by other obligations, mostly pursuant to its parent organization's commercial consulting interests and projects, the standards process sees long periods of stagnation and inactivity, after which the next effort will often redefine critical aspects of its organizational priorities, again without significant community input. Aside from this closed and undemocratic leadership nucleus, there is no provision for the community to continue the progress of standardization, and the community must simply wait for the leadership to return to the work, as the documents critical to the progress of this process are kept under the tightly-curated control of the organization's leadership.

Fourth, we must consider the Osborne effect: by taking hostage the standardization process, the MDC sends a message to current members of the community that what they currently have (the 1995 edition of the standard) isn't good enough, and to M implementers and their customers that innovations they introduce in the intervening period may be rendered obsolete by standards that may broadly change the syntax and/or semantics of such language innovations. This is a chilling effect, and one for which the community must hold the MDC directly responsible.

What, then, is an alternative way forward?

As a community, we should demand better. We should demand a share of standardization power equal to the power of MDC leadership. We should demand fair and free elections of MDC leaders, both before and after the organizational direction is set. We should demand a new, bold, and innovative standard.

M can shine on its own merits. Let's move forward and make them obvious to the rest of the world, free of bureaucratic corruption and undemocratic organizational stagnation.

Let's move forward.

Since 2014, I have been the maintainer of the primary fork of the FreeM implementation of the M programming language and database. I thought I would take some time to share some of the history of FreeM, as well as its current status and goals.

How I got involved in M and FreeM

My mentor in computer programming and UNIX was Larry Landis, who got involved heavily in the M/MUMPS programming language ca. 1991. He hyped up the M language to me from 1991 forward, and first demonstrated FreeM to me in August 1998. In 2010, I incorporated my company, Coherent Logic Development, learned M, and began doing contract work in M through Larry's company, Fourth Watch Software.

Larry was the owner of FreeM's SourceForge repository, which had not been touched in a number of years, following Fidelity National Information Services' decision to release GT.M under a free software license. In August 2011, I downloaded the source code for FreeM and did enough work on it to get it running under modern GNU/Linux systems and posted it to the mumpster.org forums.

In 2014, Larry gave me administrator access to the FreeM SourceForge repository and transferred maintainership of the project to me.

Early History of FreeM

FreeM was developed in Germany in the mid-1990s by a developer who went by the pseudonym “Shalom ha-Ashkenaz”, whose actual identity remains unknown, though it is thought by some that he is or was a dentist who learned C and developed FreeM on his own time. Shalom developed FreeM at a time when Terry Ragon of InterSystems (the company that developed the ISM implementation of M) was buying up all of his competitors and shutting them down. Shalom wished to provide a community-driven, open-source implementation of M as a bulwark against the growing threat of single-vendor hegemony over the M language. Its design—as well as some of the documentation included with the original sources—indicate that FreeM was originally targeted to the MS-DOS family of operating systems. It made use of a very limited subset of the C library, and included instructions for renaming the MS-DOS style 8.3 filenames in order to compile under UNIX.

At one point in FreeM's early history, Shalom ported FreeM from MS-DOS to SCO UNIX, the SVR3-based descendant of Microsoft XENIX, now known as SCO OpenServer—a platform still supported by FreeM. This port brought support for the scoansi terminal type, including colors and X.364 control mnemonics.

Enter the GUMP

Around the time Shalom ha-Ashkenaz was developing FreeM, Richard F. Walters, a professor from U.C. Davis, conceived of the GUMP, an acronym standing for “Generic Universal M Project”. The GUMP, following the object-oriented programming craze of the 1990s, was intended to be a toolkit allowing M implementations to be built from discrete components with a well-defined and well-specified public interface among these components. These components included the global handler (supplying the database functionality), and the interpreter/compiler (responsible for implementing M language commands). The components would have been able to communicate over a network, or in-process on the same host, enabling distributed computing functionality.

Although the specification for the GUM interface to global handlers attained a reasonably well-specified level of completeness, and Larry Landis and others developed a mostly-complete implementation of a GUM global handler, none of the other envisioned components were ever completed, and specifically, the interpreter component was missing.

Shalom's Gift

In July of 1998, Shalom ha-Ashkenaz donated the FreeM source code (then known as FreeMUMPS) to the M User's Group-Deutschland (MUG-D), hoping that the community would take the nascent implementation from its infancy through to a state of production-ready completeness and robustness. Shalom also placed a few conditions on his gift: a public release could not be made until a substantial set of milestones were reached. Per his conditions, the FreeMUMPS project must:

  • Implement the entirety of X11.1-1995
  • Use Structured System Variables instead of VIEW commands/functions
  • Raise the string size limits
  • Implement MWAPI, OMI, X11 bindings, and GKS bindings
  • Be substantially free of major bugs

Although MUG-D readily accepted the contribution of FreeMUMPS, the organization itself lacked the manpower and expertise to complete the implementation. Just as it is now, the intersection of M community members who know enough of the M language and C language to work on a project this ambitious was quite small.

Merging GUMP and FreeM

Very shortly after the contribution of FreeMUMPS to MUG-D, Richard F. Walters and a small team of developers and administrative staff who had been working on the GUMP assumed maintainership of the FreeMUMPS source code. This included representatives from the M Technology Association (an M vendor association having several foreign branches), the M Development Committee (the M standards organization hosting the ANSI/ISO standards for the M language, then sponsored by the M Technology Association), and others. The goals of this team were to:

  • Meet Shalom's requirements for a public release of FreeMUMPS
  • Convert FreeMUMPS into the first interpreter component of the GUMP

During this era, Ronald L. Fox of Hawaii (who passed in 2010) ported FreeMUMPS from SCO UNIX to Red Hat 5 and glibc-6. Steve “Saintly” Zeck also attempted to rewrite the symbol table code to lift string size limits, David Whitten enhanced some of the implementation-specific extensions, and Larry Landis integrated Saintly's symbol table work.

Early on in the GUMP maintainership of FreeM, the name of the implementation was changed from FreeMUMPS to Public Standard M, a change which was changed to Free Standard MUMPS and then FreeM when it was discovered that the PSM acronym was already in use for Patterson & Gray's M implementation. Dr. Walters also received the implementation ID of 49 from then secretary of the M Development Committee, Don Piccone.

One of the contributors to FreeM at this stage—mainly in the area of M vendor routines—was Axel Trocha, who would go on to develop and maintain his own private fork of FreeM.

The GT.M Free Software Release

GT.M, standing for Greystone Technology MUMPS, is an M implementation that was released by Greystone Technology in 1986. Greystone was later acquired by Sanchez Computer Associates, which was in turn acquired by Fidelity National Information Services.

When GT.M was released under a free software license in 2000, it seemed to negate the entire raison d’etre for FreeM, as GT.M was a well-established, robust, and high-performance M implementation with which FreeM could not then compete. Unfortunately, at this time, the GUMP and FreeM projects lost all of their momentum, and new development along these lines rapidly ceased. The final GUMP team release of FreeM was 0.6.0. However, Axel Trocha’s private port would continue.

Axel Trocha's FreeM Fork

After FreeM's momentum ceased within the primary branch of development under Richard F. Walters' leadership, Axel Trocha, an aforementioned contributor of M vendor routines and member of Dr. Walters' team, continued development on the FreeM source code. Axel added many interesting features to the FreeM codebase, including:

  • A native port to Microsoft Windows
  • Compiling FreeM as an Apache web server module, allowing FreeM to be used easily for web development
  • The ability to output HTML code in a heredoc-style format, with any line of code beginning with a left angle bracket being interpreted as HTML with support for interpolated M locals and globals
  • Extensions allowing FreeM to be used as a command-line shell, along the lines of UNIX bash, Windows cmd.exe, etc.

Axel also maintains ownership of the freem.net Internet domain, and continued issuing public releases of his FreeM port on that site until sometime after 2003, at which point he took his port entirely private. Currently, freem.net is a blank page. However, Axel's fork of FreeM continues to this day as the back-end database and programming environment for the www.elvenrunes.de website. I have communicated with Axel occasionally.

Resuming Primary Development Branch

In 2011, I downloaded the FreeM source code from the GUM Project's SourceForge repository—dormant since 2000—and updated it just enough that it would compile and run on modern GNU/Linux systems. I also quickly updated FreeM to support terminal sizes larger than 80x24.

Taking Maintainership

In 2014, Larry Landis gave me administrator access to the GUMP repository, transferring maintainership of the primary branch of FreeM development to me. Since then, I have added many features and corrected many bugs, including:

  • Adding support for proper namespaces, configured through /etc/freem.conf, which standardizes routine and global storage locations
  • Adding support for Structured System Variables
  • Adding support for the asynchronous event specification from the MDC Millennium Draft Standard
  • Adding support for constants through the CONST keyword
  • Adding a WITH keyword that allows you to specify an implicit prefix to all subsequent variable references
  • Adding a runtime watch command (WATCH), which tracks changes to specified locals or globals
  • Adding an ASSERT command, which will fail with an error message if the following expression evaluates false
  • Adding support for operators such as ++, —, +=, etc.
  • Removing the Steve “Saintly” Zeck symbol table implementation, which was unreliable, and reverting to Shalom's original implementation
  • Adding support for the GNU readline library, with persistent command line history and editing, as well as some level of command-line completion
  • Adding REPL-like functionality (in direct mode, any M expression beginning with a number will be prepended with an implicit WRITE)
  • Adding transaction processing (a work in progress)
  • Adding KVALUE and KSUBSCRIPTS
  • Adding support for the M Windowing API (MWAPI), which is also a work in progress
  • Adding the fmadm command-line utility, for database administration functions
  • Adding support for after-image journaling and forward database recovery
  • Adding support for TCP and UDP client sockets, for both IPv4 and IPv6
  • Writing a texinfo manual, from which the HTML manual is derived
  • Porting to Solaris/SPARC, Solaris/x86, Linux/s390x, Linux/armv6l, Linux/armv7l, SCO OpenServer 5.0.7, Tru64 UNIX/alpha, AIX/ppc, Mac OS X/x86, GNU HURD, Cygwin, NetBSD, FreeBSD, OpenBSD, and WSL1/2

I have also created the https://freem.coherent-logic.com website, where distribution downloads and documentation are readily available.

Future

FreeM is moving towards being a client-oriented desktop M implementation, for developing graphical user interfaces that will run on mobile and desktop devices.

I also intend to adopt the original vision of the GUMP team, dividing FreeM's functionality into discrete components having a well-specified public interface, with the ability to run in distributed computing environments over a network.

FreeM's mission is to advance the state-of-the-art in M implementations, and push the evolution of the language forward. Maintaining portability to as many vintage and modern UNIX systems as possible is held as a high priority, while portability of M routines and MDC standards compliance will be maintained only to the extent that it does not conflict with the primary goal of elegantly advancing the state-of-the-art and finding new audiences for the concepts originated by Neil Pappalardo and Octo Barnett in 1966.

FreeM is also strongly committed to free software principles, and is firmly aligned with the goals of the GNU Project and the Free Software Foundation, believing that the ethical concerns surrounding proprietary software are at least as important as the practical concerns of “open-source”.

FreeM is also being developed as a tool for enabling application development in worker/tenant cooperatives, and is committed to social justice and progressive principles.

If you are interested in FreeM, please see [https://freem.coherent-logic.com][https://freem.coherent-logic.com] for more information.

FreeM implements synchronous event handling as defined in ANSI X11.6-1995 (MWAPI) and asynchronous event handling as proposed in MDC Type A extension proposal X11/1998-28, with several significant vendor-specific extensions. Though the M Development Committee’s use of the terms “synchronous” and “asynchronous” are technically correct, the MWAPI and X11/1998-28 event handling models’ use of the terms may seem somewhat unusual or foreign to those accustomed to event handling in world-wide web technologies, such as JavaScript. The remainder of this article will explore in some depth the X11/1998-28 and MWAPI event handling models as well as the architecture with which FreeM implements and extends them.

Synchronous Events

In M parlance, a synchronous event is one originating from a graphical user interface defined in the M Windowing API (MWAPI). To begin accepting and processing synchronous events, normal, procedural M code must execute the ESTART command, which implicitly enters an event processing loop. ESTART will block the flow of M code execution on the code path in which ESTART was invoked: M code immediately following ESTART will not execute until a synchronous event handler subroutine outside the primary code path of the application calls ESTOP to stop the implicit event processing loop.

Synchronous event handlers are typically registered in the ^$WINDOW structured system variable. The following code will create the window with ID myWindow, register CLOSE^WINDOW as the event handler for the CLOSE event class on window ID myWindow–called when the close gadget is pressed or the window is closed by other means. It will then begin the implicit synchronous event processing loop:

WINDOW ;
  SET W("EVENT","CLOSE")="CLOSE^WINDOW" ; Create a window definition
  MERGE ^$WINDOW("myWindow")=W ; After this MERGE, the window will appear
  ESTART ; This enters the implicit event processing loop
  QUIT ; Will not execute until CLOSE^WINDOW calls ESTOP
  ;
  ;
CLOSE ;
  ESTOP ; Stop synchronous event processing
  QUIT

Other metadata about the CLOSE event, including the window ID and window type (among others) would be supplied to CLOSE^WINDOW by populating nodes of the ^$EVENT structured system system variable, which is implicitly NEWed prior to the relevant nodes being populated and CLOSE^WINDOW being invoked.

In FreeM, the ESTART event processing loop for the above code sample takes the following steps:

  • Check if ESTOP has been called. If so, exit the event processing loop and proceed to the next command following ESTART.

  • Wait for GTK to have a user interface event in its own internal queue. During this step, the FreeM thread in which the event loop runs goes to sleep.

    • At this point, the user closes window myWindow.

    • FreeM checks the received GTK event information against FreeM’s table of windows, and see if a CLOSE event handler has been registered for this window ID (myWindow).

    • If so, FreeM will implicitly execute NEW ^$EVENT, populate it with metadata about the event and window from which the event originated, and then execute the M subroutine specified at ^$WINDOW(“myWindow”,”EVENT”,”CLOSE”). When that subroutine (in this case, CLOSE^WINDOW) exits, FreeM will return to the top of the event processing loop (step 1).

    • If not, ignore the event and return to step 1. In the above case, this does not apply, as CLOSE^WINDOW was defined as an event handler for event class CLOSE on window ID myWindow.

The above example illustrates how, from the perspective of ESTART, this type of event processing is indeed synchronous. However, while ESTART is in control, user interface events are still processed asynchronously by the underlying windowing system. This can be confusing, as MWAPI events ride the wire between low-level and high-level concepts, requiring the developer to be at least somewhat familiar with both.

MWAPI–and therefore synchronous events in M–preceded the development of the asynchronous events specification, and unlike asynchronous events, are codified in published and existing MDC standards: specifically, ANSI X11.6.

Asynchronous Events

From the perspective of the M Development Committee, asynchronous event processing exists only as a Type A Extension–specifically, extension X11/1998-28. This extension was proposed by Arthur B. Smith in September 1996 and elevated to Type A extension status–as document X11/SC15/1998-6–in June 1998 at an MDC meeting in Boston, MA. As of this writing, FreeM is the only implementation known to have implemented any part of this proposal. Event Classes

Each asynchronous event is broadly categorized into one of several event classes, referred to as evclasses in relevant standards. FreeM event classes are as follows:

Event Class Description
COMM Allows application code to respond to communications events
HALT Allows applications to handle HALT events
IPC Supports inter-process communication
INTERRUPT Allows applications to respond to operating system interrupt signals
POWER Intended to allow applications to respond to imminent power failure messages from uninterruptible power supplies
TIMER Supports the asynchronous execution of an M subroutine after a specified time has elapsed
TRIGGER (non-standard) Allows an M subroutine to run when data in an M global is accessed, changed, or deleted
USER Designed to support user-defined events
WAPI Reserved for MWAPI events–MWAPI only supports synchronous event processing at the time of writing

Event Identifiers

Beyond the event class, events are further categorized into specific event identifiers, referred to in relevant standards as evids. Event identifiers are often used as a sort of sub-type within a particular event class. Therefore, a particular, specific event is identified by the pairing of its event class and its event identifier.

In short, event classes indicate broad categories of events, while event identifiers indicate specific types of events within an event class.

Registering Asynchronous Event Handlers

Registering an event handler is the mechanism by which the M programmer associates an event class and event identifier with an M subroutine that the M implementation will execute when that event occurs. For example, if we wanted to run the RESIZE^ASNCDEMO M routine any time the user’s terminal window was resized, we’d want to handle an event with event class INTERRUPT, and event identifier SIGWINCH. The following code will associate the above event class and identifier with the RESIZE^ASNCDEMO subroutine:

ASNCDEMO ;
  SET ^$JOB($JOB,"EVENT","INTERRUPT","SIGWINCH")="RESIZE^ASNCDEMO"
  QUIT
  ;
RESIZE ;
  WRITE "The terminal was resized!",!
  QUIT

Much like synchronous events, metadata about asynchronous events–if any such metadata exists–is populated in the ^$EVENT structured system variable. As an explanation of all possible subscripts and values of ^$EVENT is far beyond the scope of this article, you are encouraged to consult your M vendor’s documentation for more information. As of this writing, that would mean consulting the FreeM manual: no other known M implementation has yet implemented this Type A extension.

Starting and Stopping Asynchronous Event Processing

Though the action of the code above will associate an M subroutine with an event class and identifier, this alone will not cause the M implementation to begin processing asynchronous events. Much like ESTART begins processing synchronous events, ASTART must be run before asynchronous event processing can occur. The ASTART command looks like this:

ASTART:postcondition [[evclass,...] | [(evclass,...)]

As is typical with M commands, ASTART supports argumentless, inclusive, and exclusive forms. In its argumentless form, ASTART will begin asynchronous event processing for all event classes. In its inclusive form, ASTART will begin asynchronous event processing for only the specified event classes. Finally, the exclusive form of ASTART begins asynchronous event processing for all event classes except those specified.

Let’s further flesh out our ASNCDEMO routine to enable asynchronous event processing for the INTERRUPT event class:

ASNCDEMO ;
  SET ^$JOB($JOB,"EVENT","INTERRUPT","SIGWINCH")="RESIZE^ASNCDEMO"
  ASTART "INTERRUPT"
  QUIT
  ;
RESIZE ;
  WRITE "The terminal was resized!",!
  QUIT

While the above code will definitely enable asynchronous event processing for INTERRUPT events, the user would never see any output from the event handler, as the program would quit prior to any event occurring: unlike ESTART for synchronous events, ASTART is always non-blocking. Therefore, in the above example, ASTART “INTERRUPT” will enable asynchronous event processing for INTERRUPT events and return immediately. As the next command in the routine is QUIT, the routine will immediately exit. The non-blocking nature of ASTART is a primary reason why asynchronous events in M are so named: they do not block the primary code path or enter an implicit event loop.

Due to the non-blocking nature of ASTART, asynchronous event processing in M probably makes the most sense for applications that provide their own loop: for instance, an application that displays a menu, accepts a selection, performs processing, and then re-displays its menu, or, an application that runs in an I/O loop gathering data, processing it, and storing results.

Blocking and Unblocking Asynchronous Events

Each asynchronous event class is paired with an event block counter specific to that event class. This counter is a simple integer, and when nonzero, rather than executing the M subroutine associated with that event class, when an event of that class occurs, it will instead be queued for later processing. This mechanism is implicitly employed on invocation of an event handler subroutine: prior to entering the event handler, the event block counters for all event classes are incremented by one, ensuring that the execution of one event handler can never be interrupted by the execution of another. Similar to M’s incremental LOCK, event blocking is also incremental, as the block counter for an event’s event class must be zero in order for its event handlers to execute.

Event blocking and unblocking can also be achieved manually via the ABLOCK and AUNBLOCK commands, whose syntax are thus:

ABLOCK:postcondition [[evclass,...] | [(evclass,...)]
AUNBLOCK:postcondition [[evclass,...] | [(evclass,...)]

In their argumentless forms, ABLOCK and AUNBLOCK will increment or decrement the event block counters for all event classes. In their inclusive forms, they will increment or decrement the event block counters for only the specified event classes. In their exclusive forms, they will increment or decrement the event block counters for all event classes except those listed.

Remember earlier, when we mentioned that an argumentless ABLOCK is implicitly executed prior to entering an event handler subroutine, in order to prevent asynchronous event handlers from interrupting each other? Although not a feature for either the faint of heart or those without exceptionally sharp minds for writing reentrant code, it is possible (though not generally recommended) to AUNBLOCK one or more event event classes inside of an event handler to enable such reentrant behavior. The pitfalls and risks to logical integrity of M globals are so great that you should do so only with a preponderance of caution and prodigious and careful use of LOCK around global variable accesses in such event handlers: here there be dragons! FreeM Extension: System-Wide Asynchronous Events

In FreeM, the X11/1998-28 extension has been extended to support events that will be recognized in all FreeM processes on the system, rather than being limited to the current process only. The only difference is in the registering of event handlers: rather than registering handlers in ^$JOB($JOB,”EVENT”,…), system-wide event handlers are registered in ^$SYSTEM(“EVENT”,…). FreeM Asynchronous Event Handling Architecture

FreeM employs an event queue for asynchronous events, shared across all event classes. External signals from the operating system will interrupt the flow of the FreeM C code, calling callbacks internal to FreeM that will enqueue an event, along with its event class, event identifier, and metadata, in the event queue, and interpreter execution will resume. If the event is not important enough to immediately interrupt the interpreter, the event queue will be checked and handlers run after the current M command completes. If the event is extremely important, FreeM will raise error condition ZASYNC. Once ZASYNC is raised, at the next checkpoint where FreeM checks for an error condition, the internal error handler will be invoked. When $ECODE is ZASYNC, FreeM will immediately de-queue the event queue and execute all pending event handlers prior to resuming normal program execution.