Skip to Content

Assessing DH for promotion reviews

Tuesday, March 30th, 2021

Cristanne Miller
Email: ccmiller@buffalo.edu
Institution Affiliation: University at Buffalo SUNY Department of English and Arts Management

Does your department or institution have a good policy for reviewing digital projects as part of a promotion / evaluation file? Could you share that language or thoughts about good language for this purpose? Online sites mostly have guidelines putting the onus for appropriate evaluation onto faculty or making general suggestions for evaluation processes. I’m looking for policy language, e.g., “digital online site construction or collaboration counts as equivalent to print publication”–when? how? to what extent? Specifics about how it should be evaluated?

Responses by:
Erik Seeman
Eric Robert Hayot
Mary Loeffelholz
Jeff Good
Mishuana Goeman
David Chinitz
Kathleen Fitzpatrick
Cristanne Miller
Richard Griscom


Response by Erik Seeman

Email: seeman@buffalo.edu
Institutional Affiliation: University at Buffalo SUNY
Position: Chair, Department of History

A few years ago the History Department added the paragraph below to its tenure and promotion standards. It’s a very modest step toward recognizing the increasing importance of digital scholarship in the discipline of History. I hope that in the future we can add stronger language about counting digital work as part of a faculty member’s scholarly output.

“In light of the increasing visibility of digital history, the department agrees with the American Historical Association’s recommendation that “scholars who take a strong interest in digital media … should be evaluated in terms of their overall ability to use sustained, expressive, substantive, and institutional innovation to advance scholarship” (Guidelines for the Professional Evaluation of Digital Scholarship by Historians, June 2015). In particular, the department supports the AHA’s recommendations that 1) digital scholarship be evaluated in its native medium and not printed out for promotion dossiers, and 2) for candidates with a significant digital history profile, at least one of the external evaluators should be a specialist in digital history.”


Response by Eric Robert Hayot

Email: euh2@psu.edu
Institutional affiliation: Pennsylvania State University
Position: Professor, Comparative Literature

As with English at Penn State, we have no specific guidelines for evaluating DH work. Partly I think that’s because we have really not hired in these fields. But your query highlights the real need for someone or a group of someones to develop a set of guidelines that could really work, something that would create a model from which people could deviate.

The easiest way to do this is to convince some organization (HASTAC?) to create a committee. The committee proposal could/would then be voted on by the membership. Obviously, as part of such an institutional committee, there could be sharing of drafts/ideas with other larger groups.

I do think the challenge is to come up with specifics –rather than the kind of vague and general guidelines that already exist. And of course the particular challenge is to do so in a changing DH landscape—imagine if these had been written 10 years ago, what they would and would not include.


Response by Mary Loeffelholz

Email: m.loeffelholz@northeastern.edu
Institutional affiliation: Northeastern University
Position: Professor, English

At Northeastern, the Tenure and Promotion policy for the English Department includes relatively specific language about the range of possibilities for considering DH work and recently two candidates have received promotion with files that gave significant credit to their digital work. The relevant paragraphs for a tenure review evaluation are:

  • English studies comprises multiple disciplinary areas, including literary studies, rhetoric and composition, digital humanities, linguistics, and creative writing, and many English department faculty do research that is interdisciplinary or is in newly emerging fields. Modes of disseminating and evaluating scholarship and creative work vary depending on the norms of the field and the nature of the project . . .
  • The department recognizes that scholarly or creative careers may develop along a variety of paths leading to different profiles meriting tenure. For many scholars, the single-authored monograph, published by a university or other academic press with a strong reputation in the particular field, is the primary embodiment of a scholarly agenda. For others, the significant scholarly project is embodied in other formats, such as a substantial body of articles published in peer-reviewed journals prominent in the field(s); a collaboratively authored monograph or substantial body of collaboratively authored articles in which the candidate has taken a leading role; or the creation of a digital scholarly edition or creative work, curated digital archive, or digital research database.
  • In all cases, quality of achievement is demonstrated by the publication of work in venues that are respected in the particular discipline or genre; that use a peer or juried review process or equivalent selective process to evaluate the quality of work; that exercise editorial discretion; and that make the work available to appropriate audiences.

Response by Jeff Good
Email: jcgood@buffalo.edu
Institutional Affiliation: University at Buffalo SUNY
Position: Chair of Linguistics Department at UB, co-director of Socio-Spatial Approaches to the Analysis of Multilingualism

In Linguistics, we don’t have any special language about this in our tenure and promotion guidelines, and I think this would be hard to do given the diversity of publication models in the field, which pretty much spans all the well-known possibilities.

In one subfield, language documentation, in addition to producing articles and books (e.g., a full descriptive grammar of a language), scholars may also collect extensive audio and video recordings and create annotations on these recordings, all of which are curated for long-term archiving. This curation is very time intensive, and an archival corpus with extensive annotation is considered to be a very valuable resource. Scholars in this subfield have tried to develop ways to get this work recognized within the “publication economy.” For example, one journal invites those who have created archival deposits to submit papers describing their collections. These papers are then peer reviewed, with the expectation that the reviewers will also verify that the description matches what is available in the collection. Related to this are efforts to create citation standards linguistic datasets (see, e.g., https://www.rd-alliance.org/group/linguistics-data-ig/outcomes/tromsø-recommendations-citation-research-data-linguistics). When I have reviewed tenure cases where a significant part of the scholar’s output involves archival materials, I have tried to give my best estimate of a “publication equivalent” of their archival deposits. For instance, I might evaluate a medium-sized deposit as being roughly equivalent to a journal article, while a larger one could be comparable to a monograph. For a candidate being evaluated for tenure and promotion, I think the department’s guidelines matter less than whether the chair understands the need to find external reviewers who are able to make such assessments.

For digital projects that can’t clearly be labeled “research,” it is possible in language documentation work to get that to count for tenure, I think, but it would be hard to justify unless the project was oriented around a speaker community. I think a public-facing digital project would have to wait until after tenure.

The kind of “tenurable” project I have in mind might involve, for instance, a linguist creating a resource to support the maintenance of a Native American language. In some cases, doing this work might be a condition for doing research with that community in the first place. In others, it may be that the situation with the language is so critical that it would be irresponsible to delay community support projects until after tenure. Senior scholars understand these issues and can review the dossiers accordingly, assuming that the senior faculty in someone’s department know the kinds of people who should be asked to review the dossier.

I think it would be hard to write these eventualities into institutional tenure guidelines. What’s happened in language documentation is that there’s been a multi-decade international conversation around these issues that has helped people come to a shared understanding of the issues. Global-scale language endangerment was a good motivator for this.


Response by Mishuana Goeman (Tonawanda Band of Seneca)
Email: goeman@gender.ucla.edu
Institutional Affiliation: UCLA
Position: Professor American Indian Studies, Gender Studies and founding co-director of Carrying Our Ancestors Home and Mapping Indigenous LA.

My UCLA colleague David Shorter developed the Wiki Indigenous language programs http://wil.cdh.ucla.edu/ and has written in this area. As tribal Nations move to the digital for their language teaching and dictionaries this will be necessary realm to assess! Mukurtu, for instance, has enhanced its platform for this purpose and in response to tribal requests. https://mukurtu.org/​ There is a component where the language student can be offline given the poor infrastructure on reservations and digital divide.

Here is David’s Take on Digital projects: https://ethnomusicologyreview.ucla.edu/journal/volume/16/piece/459

David also suggested naming the role archivist and curator as those are categories often understood by personnel committees. His website is: https://www.davidshorter.com/


Response by David Chinitz
Email: Dchinit@luc.edu
Institutional Affiliation: Loyola University Chicago
Position: Professor, English; founding co-director Modernist Networks (ModNets)

In English at Loyola, we have no “specific language” for evaluating digital projects so the assessment depends on the nature of the project. If it has an obvious print equivalent, we treat it no differently from its equivalent in print–for instance, an article published in an online journal is counted exactly like an article published in a print journal. If the project has no print equivalent, the faculty member and chair work out a comparison based on its scope and the work involved in creating it. A digital project might be the approximate equivalent, in that sense, of a monograph, an article, or something in between. To use myself as an example, my volume of Eliot’s WWII-era prose (published digitally by Johns Hopkins) counted as a major print edition, that is, equivalent to a monograph, while co-directing ModNets counted as the equivalent of an essay collection. Incidentally, we also don’t take any points off for collaboration. The other co-director, Pamela Caughie, and I both received full credit for ModNets.


Response by Kathleen Fitzpatrick
Email: kfitz@msu.edu
Institutional Affiliation: Michigan State University
Position: Director of Digital Humanities and Professor of English, former Executive Director and Director of Scholarly Communication of the Modern Language Association

I’ve been invested for quite some time in questions about how digital scholarship is assessed in a range of evaluative contexts, including tenure and promotion reviews, and have had the opportunity to participate in producing a number of statements on the matter. One thing that the teams I’ve worked with have always tried in those processes — not always successfully — is to avoid relying on equivalencies within existing schemas, which wind up leaving the book relatively undisturbed as the object to which other things are compared. We’ve also run into ongoing challenges about how specific language about the process of evaluation for digital scholarship can or should be, given the enormous differences among projects (in terms of aims, scope, support, and more) and given the ongoing changes that new experiments, technologies, and modes of working will produce. That’s one key reason the 2012 revision of the MLA guidelines reads as radically open-ended: we were working to develop principles that could be applied without regard to the specifics of a project’s situation, and we wanted them to be as future-proofed as possible.

A couple of years ago, the English department here at MSU undertook the process of revising its bylaws, including its evaluation and RTP criteria, which are in section 4.2.1.3. We tried really hard to capture both the range of new possibilities for what scholarship can be and how it can come into being – wrestling with what “peer review” means for documentary films, exhibits, and performances, as well as for digital projects – and we struggled to avoid the re-centering of the book as the point from which equivalencies are established. What we ended up with isn’t perfect, but it’s expansive, and I particularly like the idea (in 4.2.1.3.1.3.1!) of projects that make “sustained and/or substantial contributions” to scholarly work as the category that’s at the top of the heap.

In an ideal world, however, I’d like to see a far more thoroughgoing upending of the categories of evaluation, allowing us to get rid of the research/teaching/service boxes and instead think far more fluidly about what a wide range of contributions to scholarly life and community can achieve, but that’s a larger project…


Response by Cristanne Miller
Email: ccmiller@buffalo.edu
Institutional Affiliation: University at Buffalo, SUNY
Position: Drofessor of English, Co-director of the Digital Scholarship Studio and Network; founding director of the Marianne Moore Digital Archive.

Many thanks to all who have responded online or permitted revised email messages to be posted here. I would like to call attention to some resources many of you may already know about but that may not be familiar to some—garnered from several email conversations.

I’s also like to encourage all with concerns about this topic to contribute directly to this online discussion so that I am not in the position of (as it were) curating this conversation from remarks I receive privately. dhnetworks@buffalo.edu We at the DHN/W are particularly concerned to know ways in which such assessment issues affect the promotions of people of color or LGBTQ faculty.

Guidelines for assessment of digital scholarship in promotion reviews (please add to this list!):

  1. The Association for Computers and the Humanities (ACH) https://ach.org/ach-guidelines-for-assessment-of-digital-scholarship-in-tenure-and-promotion/
  2. The Journal of Digital Humanities has an issue focusing on evaluation from 2012: http://journalofdigitalhumanities.org/1-4/closing-the-evaluation-gap/.
  3. University of Nebraska, Lincoln posts Promotion & Tenure Criteria for Assessing Digital Research in the Humanities
  4. George Mason U College of Humanities and Social Sciences (they just added language to their T&P guidelines to support digital and public humanities projects in lieu of a book):
    https://d101vc9winf8ln.cloudfront.net/documents/28581/original/CHSS_PT_Guidelines_Final.pdf?1531355841 (many thanks to Jennifer Serventi at NEH for this information!)

Posted on Humanities Commons is “The New Rigor Report,” co-authored by Kimberly Bain, Jeffrey Moro, Mariel Nyröp, and Marisa Parham: https://hcommons.org/deposits/item/hc:24463/ for the Massachusetts 5 colleges Digital Humanities, which (among other things) seeks to help departments develop language of assessment and commensurability early in each evaluative process, including at the moment of hire. Marisa Parham has also written a short article to accompany the New Rigor report: https://hcommons.org/deposits/item/hc:24461/

I have also heard that University of Oklahoma and Dickinson College have good models for “standards” language.

Everyone stresses the importance of peer review in evaluating digital work but (beyond successful grants and the publication of digital scholarship online) the resources for peer review are often not clear. Here are some journals & organizations providing peer review for digital projects and built resources (again, please add to this list if you know of others!):

  1. Reviews in Digital Humanities ed. Jennifer Guiliano and Roopika Risam
  2. Archipelagos, ed Alex Gil Fuentes –focused on Caribbean studies
  3. The Journal of Digital Humanities ed. Lisa Rhody
  4. NINES (19th-century Scholarship Online): provides peer-review of digital resources and archives created by scholars in nineteenth-century studies—British and American—for aggregation on its website
  5. ModNets: provides peer review of digital resources/archives in modernist studies, for aggregation on its site.
  6. The Society for Classical Studies publishes reviews of digital projects on its website. (Thanks to Neil Coffee for this information.)

At the advice of several people, I’ve also written to ask if the MLA would like to start a process to revise their Guidelines for Evaluating Work in the Digital Humanities and Digital Media (written in 2002, revised in 2012). With their permission, I will post what I hear from them.

Cris


Response by Richard Griscom
Email: rgriscom@gmail.com
Institutional Affiliation: Leiden University
Position: Postdoctoral Research Fellow

Thanks everyone for a very stimulating discussion. I just want to share some information about an upcoming special collection at the Journal of Open Humanities Data which explores this topic within the context of language documentation (full call for submissions available here: ;https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fl.linklyhq.com%2Fl%2FNW3J&data=04%7C01%7Cevanmori%40buffalo.edu%7C7bf0adf22b6840f3301208d91541f38f%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C637564195255698950%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BFd3Qj19PLE7cf2kvhzcDUehLzf2GSK2oW1rGmapVj0%3D&reserved=0).

In a similar fashion to the Journal of Digital Humanities issue from 2012 which Cristanne Miller mentioned, I imagine that the special issue or special collection model might be helpful for those who wish to promote the development of standards for the recognition and assessment of digital scholarship in their own (sub)discipline, especially when there is a diversity of perspectives on the matter. Participating in the dialogue about digital scholarship is also itself a contribution to the advancement of each discipline, and offering the opportunity to formally participate through publication provides a good way for scholars to receive recognition for their contribution.


Reply