Markdown everywhere

Think there's a little momentum behind markdown lately?

This article from Mashable is already half a year old, and lists seventy-eight (78!) tools for "writing and previewing markdown"!  And its topic doesn't even extend to some of the very interesting services that use markdown, like leanpub and draft, or any of the numerous markdown-to-slideshow toolkits out there...

I'm convinced enough that I've just completed an initial version of a tool for working with markdown extended to allow citation using canonical URN values, and converting the source to generic markdown that any of these tools can process.  When I've polished the docs a little more, I'll post here with further notes on markdown and its increasing importance for scholarly work.

Designing scholarly publications: some lessons we can take from programmers

Scholarly publication involves more than just making work accessible.  When scholars publish, they are contributing their work to the collective endeavor of the entire scholarly community.  In order for other scholars to inspect, critique, and build upon published scholarship, it must be appropriately:

•  identified
•  verified
•  structured for reuse
•  licensed for reuse

Scholars are fortunate that all of these requirements are shared by coders, who have consequently developed well established practices and tools to satisfy each of them.  The infrastructure that programmers rely on is especially significant for digital scholarship because it has been designed for automated interaction.  For humanists, the shift from creating scholarship designed for manual processing to scholarship designed to be used through the mediation of software and hardware is often an enormous challenge.  My experience working with many collaborators on the Homer Muiltlitext project (HMT) has convinced me that we can greatly accelerate the progress of our digital scholarly work by learning from decades of experience in software development.   (In follow-up posts, I'll illustrate some working examples that exploit the design of the HMT's digital publications.)

Identifying a digital publication

Any unit of publication must be clearly identified with a specific, fixed version.  For a print monograph, this might be an edition  number, and possibly also a printing;  journals are normally identified by a date (year, quarter, or other cycle) and a volume or other serial count.  A library catalog might then resolve that reference to a storage location identified by a call number.  Digital publications likewise must be uniquely identified in a system that recognizes different editions or versions, and permits automated resolution of identifiers to a storage location.

Consider what you would typically do if you were writing a Java program, and wanted to use Saxon (a library for processing XSLT).  You can specify the library by its Maven coordinates, giving its "publisher," net.sf.saxon, the name of the "publication"", saxon-dom, and an "edition," or version, number (e.g., 8.7).  You would then rely on an automated build process to retrieve a local copy from a repository that recognizes the identifier.  With repository management systems such as Nexus, scholars can use exactly the same system of Maven coordinates to make published material automatically retrievable.  The Homer Multitext project, for example, plans to use a Nexus repository to publish the project's archive of editorial work three times a year.  The publications belong to the group org.homermultitext, and will include a publication named hmtarchive;  versions will have names like 2014.1 (the first publication of the year 2014).

Verifying a publication 

One of the distinguishing features of publication is that it has undergone some form of review.  The review process evaluates work which, in principle, ought to be replicable.  Review of digital scholarship should include not only manual evaluation, but, where applicable, automated tests assessing the data.   All of us already apply an automated test whenever we run a spell checker over a text:  when we produce digital work with more complex structure than simply a stream of words, more extensive digital tests are called for, and ought to be included as part of the publication.

One valuable idea we can take from programming practice is "test-driven development."  In test-driven development, the programmer specifies an automated test before beginning to write some section of a program, and works on the program until it passes the test.  Of course in conventional scholarly work, we evaluate work in progress as we go along: we don't submit something for review that we have not thoroughly reviewed ourselves.  But applying a test-driven approach to the editorial work of the HMT has been an eye-opening experience.  Because it compels us to reckon with "minor" irregularities we might otherwise gloss over, it can expose assumptions needing more critical examination.  In the HMT, one test we apply after tokenizing our edited texts, for example, is  a morphological analysis of all lexical tokens, on the assumption that failures will represent either Byzantine orthographic practices unrecognized by our parsing system, or errors in our edition.  We were surprised to discover a third explanation:  a number of technical terms appearing in the scholia are not fact in the standard Greek lexicon by Liddell and Scott, and therefore failed to parse.  When we retroactively applied our morphological tests to sections that had been edited before we adopted a full test-driven approach, we uncovered further examples that, as isolated cases, editors had not noticed.

Structuring for reuse

Complex computer programs are possible in part because reusable units of code encapsulate solutions to individual problems.   For example, I should never again have to spend my time writing a program to translate ancient Greek from one encoding system to another, because I can rely on Hugh Cayless' Epidoc transcoding library.  Hugh's code has a clean interface:  define the system you're translating from, the system you're translating to, and then the `getString` method hands you your result.

One of the major challenges humanists need to address today is how to design APIs to digital scholarship.  What are the appropriate components or methods, and how should they be identified?  At a minimum, a scholarly publication should address this question in two ways:

1. Citations of source material should be expressed in technology-independent but machine-actionable notation.  In the absence of an alternative that fully accomplishes this, that he Homer Multitext project has developed a URN notation for texts (CTS URNs) and for discrete objects (CITE Object URNs), as well as an extension to CITE Object URNs for resolution-independent citation of regions of interest on an image.
2. If sections of the publication itself can be cited, they too should be addressable with CTS URNs based on some logical unit (and not by accidental physical features such as page numbers).

Licensing for reuse

The act of publication alienates a work of scholarship from the author in a form that others can use, and contributes it to the scholarly community.  In addition to an appropriate technological design, scholarly publications must therefore be available under an appropriate license that must allow at least non-commercial reuse.  For programmers, the leading such license is the GNU General Public License (GPL]), and this is ideal for source code included in scholarly publication;  for other kinds of digital data, the Creative Commons project has defined Attribution-ShareAlike licenses that achieve the same aim.

Highly trained attorneys around the world have contributed their time and expertise to developing these licenses, and in many instances tailoring them to the specific requirements of local legal systems, as well as translating them into a large number of languages.  The easiest part of designing your publication should be taking advantage of their work and applying one of these licenses to your work.

The APA privatizes, too

Unlike many of my colleagues and friends in Classics Departments around the US and abroad, I will not be travelling to Chicago this week for the annual meeting of the American Philological Association.  The APA continues to accept donations to a recently completed capital campaign with the goal of supporting a digital "Center for Classics Research and Teaching."  (See the description here.)  The APA claims that its center will "make high quality information about the Classical World available in accessible formats to the largest possible audience by using technology in new and exciting ways," but has never clearly addressed the fact that, as proposed, the center will include material for APA members only.

Like Elsevier and some other distributors, in other words, the APA wants to control who can read scholarly work as part of its "business model."  Like Elsevier, the APA leadership is doubtless sincere in its belief that its "business model" is paramount.  But like Elsevier, the APA winds up in a Wonderland, where, with Humpty Dumpty, we can make words mean whatever we choose.  The idea that closed-access material could be available to "the largest possible audience" is ludicrous.  In 2012, over a billion IPv4 addresses were in use, and, while difficult to estimate, the number of individual internet users is certainly much higher.  It must exceed the APA's membership by at least six orders of magnitude. (That is, the number of internet users is surely at least 100,000 times greater than the number of APA members.)

More simply, like Elsevier, the APA's plan privatizes scholarly work that should be published.  In criticizing Elsevier's business practices, I argued that

Scholarly publication in a digital world means that a work is openly accessible for others to inspect, critique, and build upon, and we should insist that in reviews for tenure and promotion, only scholarly publications meeting this definition qualify as published work.

We should hold professional organizations to the same standard.

Unfortunately, just as these essential scholarly values are often ignored in reviews of individuals for tenure and promotion, they are often likewise neglected in evaluation of funding requests from educational institutions, federal programs and private philanthropic organizations.  There is no quick or easy way to change these entrenched practices that directly oppose the basic working method of scholarship.  But I have the choice not to become a member of (and support with my membership fees) an organization that is building a system of information apartheid.

If you are at the APA this week, try to get a clear answer to a yes/no question:  will the APA's digital publications be openly accessible for others to inspect, critique, and build upon?

Elsevier does not publish: it privatizes

If you were shocked that Elsevier has apparently issued a takedown notice to the University of Calgary, you should consider auditioning for Claude Rains' role in Casablanca.   Elsevier has never hidden the fact that its business model depends on restricting access to scholarly work.  Alicia Wise of Elsevier responds to the post linked above with this question:

the business model is based largely on paid access post-publication, and if freely accessible on a large scale what library will continue to subscribe?

The question may be sincerely intended, but its logic is straight from Alice in Wonderland:  if Elsevier cannot profit by making scholarship publicly available — that is, by publishing it — then it must privatize the information, and sell access only to clients who cede to Elsevier control over who may read the scholarly work.

The intellectual roots of western scholarship reach back to ancient Greece, and the radical idea that scholarly understanding is not determined by political or social power.  (This is exemplified in the famous story of Euclid telling his patron and monarch, Ptolemy, that "there is no royal road to geometry.")  In our modern academic institutions, publication exposes scholarly work to public scrutiny, and serves in part to ensure that scholarly claims are not based on power over information.

Elsevier and others subvert this fundamental scholarly activity when they privatize scholarship, a simple fact that we obfuscate  when, with an Orwellian twist of language, we call it "publication." It is true that scholars who freely hand over their work to privatizers make the system possible, but who can blame an untenured faculty member who will be rewarded for contributing to the dysfunction?

We should instead unambiguously reiterate that scholarly publication in a digital world means that a work is openly accessible for others to inspect, critique, and build upon, and we should insist that in reviews for tenure and promotion, only scholarly publications meeting this definition qualify as published work.

How quickly would Elsevier's pool of submissions dry up if enough universities adopted and enforced such a requirement for real scholarly publication?

What humanists do

I recently stumbled across an interview with the very articulate Astronomer Royal Martin Reese that included this observation:

But the aim of science is to unify disparate ideas, so we don't need to remember them all. I mean we don't need to record the fall of every apple, because Newton told us they all fall the same way.

(The full transcript of the interview is here, under the arresting title "Cosmic Origami and What We Don't Know.")

I think that this remark really captures a quintessential difference between the natural sciences and the humanities.  Humanists, too, unify disparate ideas, but we must record each unique phenomenon that we study.  If we develop a unified view of oral poetry, for example, we will never conclude that "I'm familiar with the Iliad, so I don't have to remember the Odyssey," or "I've studied Greek poetry so I don't need to know about the Serbo-Croatian oral poetry that Parry and Lord recorded."  We don't study apples.  Recording and remembering are basic to scholarship in the humanities.

This has important implications for how we work in a digital world.  We record and remember through citation, so before anything else we must develop a sound infrastructure for citation.

markdown + criticmarkup

It's been a year since I last posted about markdown, and in that time, the number of interesting applications and services has continued to grow rapidly.  (If you use markdown and  haven't looked at leanpub yet, you  owe it to yourself to take a peek!)

I've been looking recently at using criticmarkup together with markdown, and it seems really promising.  Editorial on the iPad and both Multimarkdown Composer and Marked2 on OS X support displaying criticmarkup within your editor.

It would be nice to have equally convenient ways to automate accepting or rejecting suggested additions, deletions or changes, but the OS X system services in the  criticmarkup site's toolkit did not work correctly when I installed them, so I'm gisting a couple of perl scripts that accept or reject criticmarkup in standard input.

• gist to accept criticmarkup
• gist to reject criticmarkup

It shouldn't be a big job to turn those into system services. I've never used OS X's Automator, but maybe that would make a good afternoon project...?

The instrumentalist fallacy and academic publication

It is easy to confuse a tool with the task it is supposed to accomplish.  This does not necessarily cause problems when the tool and task are perfectly matched, but easily leads to misapplication of the tool.  A bicycle is a wonderful means of transportation, but should not be used to travel across bodies of water, for example.  I call this confusion "the instrumentalist fallacy," and I deal with it daily in my digital scholarly work.

The academic review process has instutionalized the instrumentalist fallacy in some specially harmful ways.  I read this account of a group of mathematicians who used a github repository to coauthor a book:  any one can clone their source, and offer improvements for the authors to pull into a subsequent version. One of the principal authors has an enlightening post about the process here.

Note the contrast that both the wired interview and the author's blog post point out: the authors chose an open collaborative process because it resulted in better scholarship, but understood that they would receive less professional recognition or credit for it.

The instrumental fallacy equates the instrument— the traditional publishing process — with its goal, vetting the quality of scholarly work.  Is it too radical to suggest that the way to assess the quality of a scholarly publication might be ... to read it?

See:

• the home page for the HoTT book
• the github repository for the book