Edit: I added some more embedded VoyantTools view.

I am currently holding a seminar on “DH: tools, methods, epistemology” (Humanités numériques: outils, méthodes, épistémologie [in French]). Last week, we received Pierre Ratinaud who is developping the excellent text analysis tool IRaMuTeQ (a python-based visual interface to R). Next week, we’ll go a bit further on the text analysis road. We’ll try to use a corpus made of speeches from Roy Jenkins as head of the European Commission (1977-1980).

The aim will be to analyse this corpus with three different tools: IRaMuTeQ / Paper Machines / Voyant Tools in order to understand differencies and similitudes between those three tools and to emphasize how important it is to choose a tool, and how decisive it is in obtaining research results. Choosing a digital tool is fully part of your research methods.

So I created this Roy Jenkins corpus (whose quality is questionable, but it’s for an exercise, not for true research), 74 speeches, in three forms:

• A zotero collection (you can’t access the documents),
• A text-file for Voyant Tools,
• A text-file with specific IRaMuTeQ encoding.

Those three tools are not always comparable, in the sense that they don’t have always similar functionalities (a part from wordclouds, the less usefull and most questionable dataviz).

A short evaluation of dh tools check-list

So here is an attempt to elaborate a check-list when a researcher wants to evaluate a digital tool:

Gathering information

• Is there a website about this tool?
• Did you collect and summarize information on this website?
• Is there research-articles about the tool? Did you read them?
• Does the information gathered (websites + articles) show that the tool fits your research’s goals?

Maturity/stability of the tool

• Did you find a roadmap for the tool?
• If yes, is it far from its first version (the version that will have all the functionalities the tool is scheduled to have)?
• Is it stable? (which means: trying the tool and/or going to forums / discussions lists to see what its users are telling about the tool)

Sustainability of the tool

• Since when does this toll exist?
• Is there a strong community supporting it?
• Is it open-source? (so the algorithm can be easily evaluated)
• Can you export your data and your results? What kind of export?

Sustainability of your research

• Well… did you understand how the tool works (including its algorithm)? (see first part of the check-list)
• Will the tool allow your research’s results to be verifiable and reproducible?

That’s what we will asking ourselves next wednesday with the three tools. Meanwhile, here are some uncommented results, for comparison.

Results

Voyant Tools

Paper Machines

• DBPedia Annotations
• Topic Modeling
• ngrams
• phrasenet 1
• phrasenet 2
• Chronological Wordclouds
• Large Wordcloud

IRaMuTeQ

Dendogram (classification)

Similitude analysis

Wordcloud

Comparison

What I feel when I compare those results of a text analysis with three different tools (a part from the bad quality of the OCR of some documents):

• Design is at the heart of interpretation: the way those results are displayed is influencing the interpretation of the data. It’s outrageous in the case of wordcloud – and that’s a reason why wordclouds are so questionable;
• Those three tools are rather complementary. It seems to be obvious, but it is really important not to use only one tool;
• It is easier to get back to the text and to alternate distant and close readings of your sources with IRaMuTeQ especially and Voyant Tools (a bit less than with IRaMuTeQ). It is sometimes possible with Paper Machines, but not that much. For a historian like me, it’s just strategic to be able a multi-scale reading of those sources.

---

---