Functional programming – the use and evaluation of functions as a programming paradigm – has a long and productive history in the world of programming languages. Lisp came about in the 1950s in the search for a convenient language to represent mathematical concepts in programs, borrowing from the lambda calculus of the logician Alonzo Church. More recent languages have in turn embraced many aspects of Lisp – in addition to Lisp's offspring such as Scheme and haskell, you will find elements of functional constructs in Java, Python, Ruby, and Perl. Mathematicaitself has clear bloodlines to Lisp, including the ability to operate on data structures such as lists as single objects and in its representation of mathematical properties through rules.

Mathematica functions, unlike those in many other languages, are considered “first-class” objects, meaning that they can be used as arguments to other functions, they can be returned as values, and they can be part of many other kinds of data objects such as arrays. In addition, you can create and use functions at runtime, that is, when you evaluate an expression. This functionalstyle of programming distinguishes Mathematicafrom traditional procedural languages like C and Fortran. Facility with functional programming is therefore essential for taking full advantage of the Mathematica language to solve your computational tasks.

We start with some of the most useful functional programming constructs – higher-order functions such as Map, Apply, Thread, Select, and Outer. We then introduce iteration, a mechanism by which the output of one computation is fed as input into the next.

Programming with Mathematica

In its brief history, the world of programming has undergone a remarkable evolution. Those of us old enough to remember boxes of punch cards and batch jobs couldn't be happier about some of these changes. One could argue that the limitations, physical and conceptual,of the early programming environments helped to focus that world in a very singular manner. Eventually, efforts to overcome those limitations led to a very visible and broad transformation of the world of computer programming. We now have a plethora of languages, paradigms, and environments to choose from. At times this embarrassment of riches can be a bit overwhelming, but I think most would agree that we are fortunate to have such variety in programming languages with which to do our work.

I learned about Mathematica as I suspect many people have – after using several languages over the years, a colleague introduced me to a new and very different tool, Mathematica. I soon realized that it was going to help me in my work in ways that previous languages could not. Perhaps the most notable feature was how quickly I could translate the statement of a problem to a working program. This was no doubt due to having a functional style of programming at my fingertips but also being able to think in terms of rules and patterns seemed to fit well with my background in mathematics.

Well, Mathematica is no longer a young up-start in the programming world. It has been around now for over 25 years, making it, if not an elder statesman, certainly a mature and familiar player. And one that is used by people in fields as varied as linguistics, bioinformatics, engineering, and information theory. Like myself, many people are first introduced to it in an academic setting. Many more are introduced through a colleague at work. Still others have seen it mentioned in various media and are curious as to what it is all about. After using it to do basic or more advanced computation, most users soon find the need to extend the default set of tools that come with Mathematica.Programming is the ticket.

So what makes Mathematica such a useful programming tool? First, it is a well-designed language, one whose internal logic will be quite apparent as you get to know it.

Mathematica is a large system used across an astonishing array of disciplines – physics, bioinformatics, geo-science, linguistics, network analysis, optics, risk management, software engineering, and many more. It integrates tools for importing, analyzing, simulating, visualizing, reporting, and connecting to other programs. Underlying all of these tools is a modern programming language with which you can extend the things you can do with Mathematica almost limitlessly.

This book focuses on the programming language, but before we dive into details of syntax and structure, it will be helpful to look at some of the essential elements of actual programs to give you a better sense of what it means to program with Mathematica. So, to start, let us walk through the creation of a short program so you can see how a Mathematica programmer might solve a problem. We will not attempt to write the most efficient or shortest code here, but will instead concentrate on the process of programming – things like rewording the problem, finding the right tools and approach, and testing your program. Hopefully, this prelude will give you a better sense of the breadth and depth of the Mathematica language and help you answer the question, “Why use Mathematica to write programs?”

The second aim of this chapter is to help you become familiar with the Mathematica environment, showing you how to get started, how to work with the interface, documentation, and programming tools, and how to start putting them together to do interesting things. The basic syntax used by Mathematica is introduced, including functions and lists and several alternate syntaxes that you can use to input expressions. This is followed by information on how to get help when you get stuck or have trouble understanding an error that has occurred. If you are already familiar with these aspects of Mathematica, feel free to skim or skip these topics and return to them when the need arises.

Most of the programs we have created in the first five chapters are relatively short and compact, focused on solving problems that are quite manageable. These pieces of code are often sufficient when working on a problem and you are looking for a quick result. But before long, something more is needed: a series of computations that have to be repeated perhaps, with slight variations each time; or you need to share your code with colleagues or students; or perhaps you are developing an application for a client who knows little of the Mathematica environment or language and is looking for a solution that integrates with what they are already comfortable with.

Whether you are writing code for yourself or others, it is important to develop programs that are not dependent upon the current state of the user's Mathematica session: previously defined symbols should not interfere with code in any way; side-effects of an evaluation should not cause surprises; programs should respond gracefully when things go wrong. All modern programming languages have frameworks designed to deal with these things, and Mathematica is no exception. This chapter introduces these structures and incorporates them throughout the rest of the book.

The first construct we will look at is localization of variables. This is a mechanism to isolate symbols and definitions that are local to a function in order to keep them from interfering with, or being interfered by, global symbols.

Providing options to functions and issuing messages when things go wrong are common mechanisms for getting your programs to look and behave like built-in functions; they are introduced in Section 6.2.

The chapter concludes with several larger pieces of code that put many of these pieces together in one program.

Throughout our history, humans have developed thousands of languages in order to communicate with each other. These natural languages, such as Mandarin, Bantu, or Finnish, evolved through ordinary usage over millennia, with rules for grammar developing organically and changing frequently. Then in 1879, Gottlieb Frege, in his Begriffsschrift, invented what is often referred to as the first formal language. Formal languages are designed for specific tasks; in Frege's case, he was attempting to formalize mathematical logic. Mathematics itself is a formal language, one used to express ideas about number, pattern, space, and other concepts. Although they too have evolved over many years, mathematics and logic differ from natural languages in that they have been formally constructed and have quite explicit and rigid rules regarding syntax.

Programming languages are, like mathematics, formal languages, but they are constructed to communicate with a machine and to command it to carry out certain instructions. Thought to have their origin in the 1940s and 1950s when the first digital computers came about, in fact programming languages go back much earlier to the creation of the Jacquard loom (1801), player pianos (mid-1800s), and mechanical organs (Anselmo Gavioli, 1892), all of which used physical punch cards or rolls to code and store instructions to weave fabric or play an instrument. The connection with computing probably begins with Herman Hollerith's innovation of using punch cards to store information for the US Census of 1890. Hollerith later founded a company to market his inventions and in 1924, that company became IBM.

Regardless of the mechanism used to store programs, all programming languages are described by specifying their syntax and semantics. Syntax refers to the form, indicating what symbols can be put together and in what order to make a meaningful construct in a given language. In other words, the syntax of a programming language is the set of rules that define what is a valid input or program.

When you first start to program your emphasis is usually on correctness, that is, getting your programs to run and return accurate and error-free results - and rightly so. There is little point in trying to speed up a program that returns incorrect answers! You develop your programs, prototyping with simple inputs so that you can see at a glance how things are progressing. At some point in the development process you start to increase the size or complexity of the inputs to your program and, if all goes well, the program scales well. But commonly, there are bottlenecks at various stages of the computation that slow things down; or there may be a large increase in the amount of memory needed to represent or store an expression or result. Some of these situations may be unavoidable, but often you can find optimizations that improve the efficiency and running time of your programs. This chapter introduces some of the optimization principles to think about both during the development process and after your programs are complete and you are satisfied that they produce the desired output.

How, you might ask, does one quantify efficiency? There are two measures we will focus on - timing and memory footprint. The importance of these two measures is highly subjective. Squeezing another tenth of a second out of a computation that is only going to be run once or twice does not make a lot of sense. However, if that computation is part of a loop that is going to be evaluated thousands of times, little things really start to add up. You will be the best judge of where to focus your efforts.

After you have developed several programs for some related tasks, you will find it convenient to group them together and make them available as a cohesive whole. Packages are designed to make it easy to distribute your programs to others, but they also provide a framework for you to write programs that integrate with Mathematicaseamlessly.

A package is simply a text file containing Mathematicacode. Typically you put related functions in a package. So there might be a computational geometry package or a random walks package that includes functions in support of those tasks. The package framework includes a name-localizing construct, analogous to Module, but for entire files of definitions. The idea is to allow you, the programmer, to define a collection of functions for export.These exported functions are what the users of your package will work with and are often referred to as publicfunctions. Other functions, those that are not for export,are auxiliary, or privatefunctions, and are not intended to be accessible to users. The package framework, and contexts specifically, provide a convenient way to declare some functions public and others private. In this chapter we will describe this framework and show how to write, install, and use the packages developed with it.

Working with packages

Loading and using packages

Upon starting a Mathematicasession, the built-in functions are immediately available for you to use. There are, however, many more functions that you can access that reside in files supplied with Mathematica.The definitions in those files are placed in special structures called packages.Indeed, these files themselves are often called “packages” instead of “files.”

Mathematicapackages have been written for many different domains. They are provided with each version of Mathematicaand are referred to as the Standard Extra Packages. Their documentation is available in the Documentation Center (under the Help menu) and they provide a good set of examples for learning about package creation and usage.

The idea of representing an idealized version of an object with something called a pattern is central to mathematics and computer science. For the purposes of search, patterns provide a template with which to compare expressions. They can be used to filter data by selecting only those parts that match the pattern. Because of the wide applicability of patterns, many modern programming languages have extensive pattern-matching capabilities that enable them to identify objects that meet some criteria in order to classify, select, or transform those objects through the use of rules.

Pattern matching in Mathematica is done through a powerful yet flexible pattern language. Used in rules to transform expressions from one form to another, they can be applied to broad classes of expressions or they can be limited to very narrowly-defined objects through the use of conditional and structured pattern matching. Pattern matching is the key to identifying which rules should be applied to expressions that you wish to transform.

If you have used regular expressions in languages such as Perl or Ruby, or via libraries in Java, Python, or C++, then you are already familiar with pattern matching on strings (discussed further in Chapter 7). Mathematica's pattern language generalizes this to arbitrary objects and expressions. Although the syntax may be new to you, with practice it becomes natural, providing a direct connection between the statement of a problem and its expression in a program. This chapter starts with an introduction to patterns and pattern matching and then proceeds to a discussion of transformation rules in which patterns are used to identify the parts of an expression that are to be transformed. The chapter concludes with several concrete examples that make use of pattern matching and transformation rules to show their application to some common programming tasks.

Lists are the key data structure used in Mathematica to group objects together. They share some features with arrays in other languages such as C and Java, but they are more general and can be used to represent a wide range of objects: vectors, matrices, tensors, iterator and parameter specifications, and much more.

Because lists are so fundamental, an extensive set of built-in functions is available to manipulate them in a variety of ways. In this chapter, we start by looking at the structure and syntax of lists before moving on to constructing, measuring, and testing lists. We then introduce some of the built-in functionality used to manipulate lists such as sorting and partitioning. Finally, we will discuss associations, a feature first introduced in Mathematica 10. Associations provide a framework for efficient representation and lookup of large data structures such as associative arrays (for example, a large database of article and book references or a music library).

Many of the things you might wish to do with a list or association can be accomplished using built-in functions and the programming concepts in this book. And most of these operations extend to arbitrary expressions in a fairly natural way, as we will see in later chapters. As such, it is important to have a solid understanding of these functions before going further, since a key to efficient programming in Mathematica is to use the built-in functions whenever possible to manipulate lists and associations as well as general expressions.

Creating and displaying lists

List structure and syntax

The standard input form of a list is a sequence of elements separated by commas and enclosed in curly braces:

{e1, e2, …, en}

Internally, lists are stored in the functional form using the List function with an arbitrary number of arguments.

Strings are used across many disciplines to represent data, filenames, and other objects: they are used by linguists studying representation, classification, and patterns involved in audio and text usage; biologists working with genomic data as strings are interested in sequence structure and assembly and perform extensive statistical analysis on strings; programmers operate on string data for such tasks as text search, file manipulation, and text processing. Strings are so ubiquitous that almost every modern programming language has a string data type and dozens of functions for operating on strings.

In this chapter we will introduce the tools available for working with strings, starting with a look at their structure and syntax, then moving on to a discussion of the many high-level functions that are optimized for string manipulation. Many of the functions introduced in Chapter 3 for operating on lists have analogs with strings, and you would do well to first make sure you understand those before tackling the string functions.

String patterns follow on the discussion of more general patterns introduced in Chapter 4. We will introduce an alternative syntax – regular expressions – that provides a compact and efficient mechanism for working with strings. The chapter closes with several applied examples from bioinformatics (creating random strings, partitioning strings, analyzing GC content, displaying sequences in tables) and linguistics (text processing, corpus analysis, n-grams, collocation, word and sentence length frequency).