Cristanne Miller
Institution Affiliation: University at Buffalo SUNY Department of English and Arts Management
Does your department or institution have a good policy for reviewing digital projects as part of a promotion / evaluation file? Could you share that language or thoughts about good language for this purpose? Online sites mostly have guidelines putting the onus for appropriate evaluation onto faculty or making general suggestions for evaluation processes. I’m looking for policy language, e.g., “digital online site construction or collaboration counts as equivalent to print publication”–when? how? to what extent? Specifics about how it should be evaluated?
Response by Erik Seeman
Institutional Affiliation: University at Buffalo SUNY
Position: Chair, Department of History
A few years ago the History Department added the paragraph below to its tenure and promotion standards. It’s a very modest step toward recognizing the increasing importance of digital scholarship in the discipline of History. I hope that in the future we can add stronger language about counting digital work as part of a faculty member’s scholarly output.
“In light of the increasing visibility of digital history, the department agrees with the American Historical Association’s recommendation that “scholars who take a strong interest in digital media … should be evaluated in terms of their overall ability to use sustained, expressive, substantive, and institutional innovation to advance scholarship” (Guidelines for the Professional Evaluation of Digital Scholarship by Historians, June 2015). In particular, the department supports the AHA’s recommendations that 1) digital scholarship be evaluated in its native medium and not printed out for promotion dossiers, and 2) for candidates with a significant digital history profile, at least one of the external evaluators should be a specialist in digital history.”
Response by Eric Robert Hayot
Institutional affiliation: Pennsylvania State University
Position: Professor, Comparative Literature
As with English at Penn State, we have no specific guidelines for evaluating DH work. Partly I think that’s because we have really not hired in these fields. But your query highlights the real need for someone or a group of someones to develop a set of guidelines that could really work, something that would create a model from which people could deviate.
The easiest way to do this is to convince some organization (HASTAC?) to create a committee. The committee proposal could/would then be voted on by the membership. Obviously, as part of such an institutional committee, there could be sharing of drafts/ideas with other larger groups.
I do think the challenge is to come up with specifics –rather than the kind of vague and general guidelines that already exist. And of course the particular challenge is to do so in a changing DH landscape—imagine if these had been written 10 years ago, what they would and would not include.
Response by Mary Loeffelholz
Institutional affiliation: Northeastern University
Position: Professor, English
At Northeastern, the Tenure and Promotion policy for the English Department includes relatively specific language about the range of possibilities for considering DH work and recently two candidates have received promotion with files that gave significant credit to their digital work. The relevant paragraphs for a tenure review evaluation are:
-
English studies comprises multiple disciplinary areas, including literary studies, rhetoric and composition, digital humanities, linguistics, and creative writing, and many English department faculty do research that is interdisciplinary or is in newly emerging fields. Modes of disseminating and evaluating scholarship and creative work vary depending on the norms of the field and the nature of the project . . .
-
The department recognizes that scholarly or creative careers may develop along a variety of paths leading to different profiles meriting tenure. For many scholars, the single-authored monograph, published by a university or other academic press with a strong reputation in the particular field, is the primary embodiment of a scholarly agenda. For others, the significant scholarly project is embodied in other formats, such as a substantial body of articles published in peer-reviewed journals prominent in the field(s); a collaboratively authored monograph or substantial body of collaboratively authored articles in which the candidate has taken a leading role; or the creation of a digital scholarly edition or creative work, curated digital archive, or digital research database.
-
In all cases, quality of achievement is demonstrated by the publication of work in venues that are respected in the particular discipline or genre; that use a peer or juried review process or equivalent selective process to evaluate the quality of work; that exercise editorial discretion; and that make the work available to appropriate audiences.
Response by Jeff Good
Institutional Affiliation: University at Buffalo SUNY
Position: Chair of Linguistics Department at UB, co-director of Socio-Spatial Approaches to the Analysis of Multilingualism
In Linguistics, we don’t have any special language about this in our tenure and promotion guidelines, and I think this would be hard to do given the diversity of publication models in the field, which pretty much spans all the well-known possibilities.
In one subfield, language documentation, in addition to producing articles and books (e.g., a full descriptive grammar of a language), scholars may also collect extensive audio and video recordings and create annotations on these recordings, all of which are curated for long-term archiving. This curation is very time intensive, and an archival corpus with extensive annotation is considered to be a very valuable resource. Scholars in this subfield have tried to develop ways to get this work recognized within the “publication economy.” For example, one journal invites those who have created archival deposits to submit papers describing their collections. These papers are then peer reviewed, with the expectation that the reviewers will also verify that the description matches what is available in the collection. Related to this are efforts to create citation standards linguistic datasets (see, e.g., https://www.rd-alliance.org/group/linguistics-data-ig/outcomes/tromsø-recommendations-citation-research-data-linguistics). When I have reviewed tenure cases where a significant part of the scholar’s output involves archival materials, I have tried to give my best estimate of a “publication equivalent” of their archival deposits. For instance, I might evaluate a medium-sized deposit as being roughly equivalent to a journal article, while a larger one could be comparable to a monograph. For a candidate being evaluated for tenure and promotion, I think the department’s guidelines matter less than whether the chair understands the need to find external reviewers who are able to make such assessments.
For digital projects that can’t clearly be labeled “research,” it is possible in language documentation work to get that to count for tenure, I think, but it would be hard to justify unless the project was oriented around a speaker community. I think a public-facing digital project would have to wait until after tenure.
The kind of “tenurable” project I have in mind might involve, for instance, a linguist creating a resource to support the maintenance of a Native American language. In some cases, doing this work might be a condition for doing research with that community in the first place. In others, it may be that the situation with the language is so critical that it would be irresponsible to delay community support projects until after tenure. Senior scholars understand these issues and can review the dossiers accordingly, assuming that the senior faculty in someone’s department know the kinds of people who should be asked to review the dossier.
I think it would be hard to write these eventualities into institutional tenure guidelines. What’s happened in language documentation is that there’s been a multi-decade international conversation around these issues that has helped people come to a shared understanding of the issues. Global-scale language endangerment was a good motivator for this.
Response by Mishuana Goeman (Tonawanda Band of Seneca)
Institutional Affiliation: UCLA
Position: Professor American Indian Studies, Gender Studies and founding co-director of Carrying Our Ancestors Home and Mapping Indigenous LA.
My UCLA colleague David Shorter developed the Wiki Indigenous language programs http://wil.cdh.ucla.edu/ and has written in this area. As tribal Nations move to the digital for their language teaching and dictionaries this will be necessary realm to assess! Mukurtu, for instance, has enhanced its platform for this purpose and in response to tribal requests. https://mukurtu.org/ There is a component where the language student can be offline given the poor infrastructure on reservations and digital divide.
Here is David’s Take on Digital projects: https://ethnomusicologyreview.ucla.edu/journal/volume/16/piece/459
David also suggested naming the role archivist and curator as those are categories often understood by personnel committees. His website is: https://www.davidshorter.com/
Response by David Chinitz
Institutional Affiliation: Loyola University Chicago
Position: Professor, English; founding co-director Modernist Networks (ModNets)
In English at Loyola, we have no “specific language” for evaluating digital projects so the assessment depends on the nature of the project. If it has an obvious print equivalent, we treat it no differently from its equivalent in print–for instance, an article published in an online journal is counted exactly like an article published in a print journal. If the project has no print equivalent, the faculty member and chair work out a comparison based on its scope and the work involved in creating it. A digital project might be the approximate equivalent, in that sense, of a monograph, an article, or something in between. To use myself as an example, my volume of Eliot’s WWII-era prose (published digitally by Johns Hopkins) counted as a major print edition, that is, equivalent to a monograph, while co-directing ModNets counted as the equivalent of an essay collection. Incidentally, we also don’t take any points off for collaboration. The other co-director, Pamela Caughie, and I both received full credit for ModNets.
Response by Kathleen Fitzpatrick
Institutional Affiliation: Michigan State University
Position: Director of Digital Humanities and Professor of English, former Executive Director and Director of Scholarly Communication of the Modern Language Association
I’ve been invested for quite some time in questions about how digital scholarship is assessed in a range of evaluative contexts, including tenure and promotion reviews, and have had the opportunity to participate in producing a number of statements on the matter. One thing that the teams I’ve worked with have always tried in those processes — not always successfully — is to avoid relying on equivalencies within existing schemas, which wind up leaving the book relatively undisturbed as the object to which other things are compared. We’ve also run into ongoing challenges about how specific language about the process of evaluation for digital scholarship can or should be, given the enormous differences among projects (in terms of aims, scope, support, and more) and given the ongoing changes that new experiments, technologies, and modes of working will produce. That’s one key reason the 2012 revision of the MLA guidelines reads as radically open-ended: we were working to develop principles that could be applied without regard to the specifics of a project’s situation, and we wanted them to be as future-proofed as possible.
A couple of years ago, the English department here at MSU undertook the process of revising its bylaws, including its evaluation and RTP criteria, which are in section 4.2.1.3. We tried really hard to capture both the range of new possibilities for what scholarship can be and how it can come into being – wrestling with what “peer review” means for documentary films, exhibits, and performances, as well as for digital projects – and we struggled to avoid the re-centering of the book as the point from which equivalencies are established. What we ended up with isn’t perfect, but it’s expansive, and I particularly like the idea (in 4.2.1.3.1.3.1!) of projects that make “sustained and/or substantial contributions” to scholarly work as the category that’s at the top of the heap.
In an ideal world, however, I’d like to see a far more thoroughgoing upending of the categories of evaluation, allowing us to get rid of the research/teaching/service boxes and instead think far more fluidly about what a wide range of contributions to scholarly life and community can achieve, but that’s a larger project…
Response by Cristanne Miller
Institutional Affiliation: University at Buffalo, SUNY
Position: Drofessor of English, Co-director of the Digital Scholarship Studio and Network; founding director of the Marianne Moore Digital Archive.
Many thanks to all who have responded online or permitted revised email messages to be posted here. I would like to call attention to some resources many of you may already know about but that may not be familiar to some—garnered from several email conversations.
I’s also like to encourage all with concerns about this topic to contribute directly to this online discussion so that I am not in the position of (as it were) curating this conversation from remarks I receive privately. dhnetworks@buffalo.edu We at the DHN/W are particularly concerned to know ways in which such assessment issues affect the promotions of people of color or LGBTQ faculty.
Guidelines for assessment of digital scholarship in promotion reviews (please add to this list!):
- The Association for Computers and the Humanities (ACH) https://ach.org/ach-guidelines-for-assessment-of-digital-scholarship-in-tenure-and-promotion/
- The Journal of Digital Humanities has an issue focusing on evaluation from 2012: http://journalofdigitalhumanities.org/1-4/closing-the-evaluation-gap/.
- University of Nebraska, Lincoln posts Promotion & Tenure Criteria for Assessing Digital Research in the Humanities
- George Mason U College of Humanities and Social Sciences (they just added language to their T&P guidelines to support digital and public humanities projects in lieu of a book):
https://d101vc9winf8ln.cloudfront.net/documents/28581/original/CHSS_PT_Guidelines_Final.pdf?1531355841 (many thanks to Jennifer Serventi at NEH for this information!)Posted on Humanities Commons is “The New Rigor Report,” co-authored by Kimberly Bain, Jeffrey Moro, Mariel Nyröp, and Marisa Parham: https://hcommons.org/deposits/item/hc:24463/ for the Massachusetts 5 colleges Digital Humanities, which (among other things) seeks to help departments develop language of assessment and commensurability early in each evaluative process, including at the moment of hire. Marisa Parham has also written a short article to accompany the New Rigor report: https://hcommons.org/deposits/item/hc:24461/
I have also heard that University of Oklahoma and Dickinson College have good models for “standards” language.
Everyone stresses the importance of peer review in evaluating digital work but (beyond successful grants and the publication of digital scholarship online) the resources for peer review are often not clear. Here are some journals & organizations providing peer review for digital projects and built resources (again, please add to this list if you know of others!):
- Reviews in Digital Humanities ed. Jennifer Guiliano and Roopika Risam
- Archipelagos, ed Alex Gil Fuentes –focused on Caribbean studies
- The Journal of Digital Humanities ed. Lisa Rhody
- NINES (19th-century Scholarship Online): provides peer-review of digital resources and archives created by scholars in nineteenth-century studies—British and American—for aggregation on its website
- ModNets: provides peer review of digital resources/archives in modernist studies, for aggregation on its site.
- The Society for Classical Studies publishes reviews of digital projects on its website. (Thanks to Neil Coffee for this information.)
At the advice of several people, I’ve also written to ask if the MLA would like to start a process to revise their Guidelines for Evaluating Work in the Digital Humanities and Digital Media (written in 2002, revised in 2012). With their permission, I will post what I hear from them.
Cris
Response by Richard Griscom
Institutional Affiliation: Leiden University
Position: Postdoctoral Research Fellow
Thanks everyone for a very stimulating discussion. I just want to share some information about an upcoming special collection at the Journal of Open Humanities Data which explores this topic within the context of language documentation (full call for submissions available here: https://nam12.safelinks.protection.outlook.com
In a similar fashion to the Journal of Digital Humanities issue from 2012 which Cristanne Miller mentioned, I imagine that the special issue or special collection model might be helpful for those who wish to promote the development of standards for the recognition and assessment of digital scholarship in their own (sub)discipline, especially when there is a diversity of perspectives on the matter. Participating in the dialogue about digital scholarship is also itself a contribution to the advancement of each discipline, and offering the opportunity to formally participate through publication provides a good way for scholars to receive recognition for their contribution.
Response by Sally E. Hadden (through Cristanne Miller)
Position: Associate Professor, History
I’m embedding a link here to the Western Michigan University History Department’s DPS (departmental policy statement) which contains all the definitions used for how to be tenured, promoted, and what sorts of activities are recognized. Sections II (K) and V are the spots worth reviewing. Many of the activities included on the list for scholarly recognition go beyond the monograph or article, in part because WMU has a long connection with the field of public history. . . and as a result, it recognizes grant getting, exhibitions, and project-based activities that may be in a variety of formats.
Having said that, while these definitions of what merits scholarly recognition are many steps ahead of some institutions, when viewed in the light of digital humanities projects, there is plenty of revision still to be done! Coding a project, for example, would not necessarily fall under the definitions of II (K) and V. So I offer this not as a template of what the best options might be, but as a description of what you might aim for as a way point. I would also like to see what others have in their tenure or promotion descriptions. One of the best things I’ve read is that peer-review means peer-review, regardless of medium. So it shouldn’t privilege paper over electrons when it comes to intellectual output.
Response by Katherine D. Harris (through Cristanne Miller)
Position: Director, Public Programming, College of Humanities & the Arts; Co-Editor, Digital Pedagogy in the Humanities
I made public my promotion narrative, in which I lay out how to review my digital projects: “Moving up the Professional Ladder – Promotion Narrative for Full Professor.” In my Department, we have written capacious guidelines in terms of scholarship that are binding as the dossier moves up through the various committees and administrators, accompanied by our new-ish and well written RTP guidelines (authored by our Academic Senate). We are unionized across all of the Cal State campuses, so the RTP process is highly mediated by policy on each campus. We simply roll the elements of DH projects into our department guidelines by valuing collaborative work (and other DH methods). Also, our Academic Senate approved for use this year a lengthy clause about the “scholarship of engagement” (see p.9 of the entire policy, S15-8).
I would suggest that the faculty and admins looking at DH projects rely on reviews from external experts to gauge their efficacy. Even though we (SJSU) have capacious guidelines and are opening up our RTP university policies to be accepting of many kinds of scholarship, there is always someone sitting on a committee who declares that “digital doesn’t carry the same heft as an article/book.” But, now there are so many of us who came up with different kinds of scholarship than traditional peer reviewed articles or monographs that we can mitigate that person’s influence. (Also, we have much more freedom as a master’s granting public university than a research-intensive university about the shape of our scholarship, accepting the teacher-scholar model, and articulating curricular integration of DH projects as a form of engagement.)
This is a perennial dilemma that I wish we had solved about 17 years ago, but you can see that there is a lot of work already done and even more ongoing. I think the biggest issue is to shift the culture on individual campuses….and that takes a loooooonnnnngggg time, unfortunately.