LibGuides: Text mining : Corpora and databases: categories

This list provides a selection of corpora and databases available for text mining. Because corpora and databases can change their Terms of Use, we cannot verify whether you are allowed to apply text mining techniques to the sources below. You must comply with the copyright law and GDPR which is your own responsibility. This information can often be found via the platform's Terms of Use, or by contacting the platform itself. See also the disclaimer on this LibGuide. In case you still have any questions, you can contact us at textminingsupport@ru.nl.

Text mining corpora/databases

Nexis Uni
Newspaper archive with full articles from leading newspapers in the Netherlands. These include: De Volkskrant, Het Parool, De Telegraaf, Trouw, Algemeen Dagblad, and NRC Handelsblad. Dutch and international news with full articles from hundreds of local, national, and international newspapers, magazines, news agencies, publications and international news from English and non-English sources. There is also access to Dutch and international company information.

more... less...

Downloading: bulk downloads possible of up to 500 documents at a time (PDF, Word or RTF); API not available within our license.
Chronicling America
Search America's historic newspaper pages from 1756-1963 or use the U.S. Newspaper Directory to find information about American newspapers published between 1690-present.

more... less...

API available: https://chroniclingamerica.loc.gov/about/api/
Delpher
Historical Dutch newspapers, books, journals, and radio bulletins available in full-text.

more... less...

Downloading: Individual texts are available in XML, txt and PDF formats. Bulk downloads possible via API's. More information on the website of the Royal Dutch Library (Koninklijke Bibliotheek): https://www.kb.nl/en/research-find/for-researchers/data-services-apis-and-downloads

Corpus Hedendaags Nederlands
The Corpus of Contemporary Dutch (CDH) contains more than 2,500,000 Dutch-language texts (newspapers, magazines, books, news broadcasts and blogs) from the Netherlands, Belgium, Suriname and the Dutch Caribbean. Created by the Dutch Language Institute (Instituut voor de Nederlandse Taal).

more... less...

Downloading: Invididual texts cannot be accessed in their entirety. Instead, the corpus can be searched and analysed as a whole.
Corpus Middelnederlands
The Corpus Middle Dutch is a collection of 372 Middle Dutch documents from the period of 1300-1550, made available by the Dutch Language Institute (Instituut voor de Nederlandse Taal).

more... less...

Downloading: the texts can be searched and analyzed via a web interface. It is possible to make selections of documents using filters.
Corpus Oudnederlands
The Corpus Old Dutch is a collection of Dutch language texts and fragments from the period 475-1200, made available by the Dutch Language Institute (Instituut voor de Nederlandse Taal). The words have been lemmatised and POS-tagged.

more... less...

Downloading: the texts can be searched and analyzed via a web interface. It is possible to search on word, lemma, and part-of-speech. You can make selections of documents using filters.
Digitale bibliotheek voor de Nederlandse letteren (DBNL)
The Digital Library of Dutch Literature is a digital collection of texts relating to Dutch literature, language, and culture, from many different time periods.

more... less...

Downloading: Individual texts are available in XML, txt and PDF formats. The complete collection of public domain titles is available as a downloadable dataset.

Project Gutenberg
Online library of free books in digital format, mainly older works of which copyright has expired, such as literary classics and other historical texts.

more... less...

Downloading: individual downloads available in many file formats, including plain text files. Bulk downloads possible with the free tool wget.
TDM information: https://www.gutenberg.org/policy/robot_access.html
HathiTrust
HathiTrust is an online archive of digitalised materials from (mainly American) academic libraries. A substantial part of it is freely accessible due to expired copyright or an open access license.

more... less...

Downloading: Some books can be downloaded completely in one go, others can only be downloaded page by page. However, bulk access and text mining functionality is possible through HTRC analytics.
TDM information: https://analytics.hathitrust.org/
Standard Ebooks
Standard Ebooks is a volunteer-driven project that produces new editions of public domain ebooks that are lovingly formatted, open source, free of U.S. copyright restrictions, and free of cost.

more... less...

API not available.
Delpher
Historical Dutch newspapers, books, journals, and radio bulletins available in full-text.

more... less...

Downloading: Individual texts are available in XML, txt and PDF formats. Bulk downloads possible via API's. More information on the website of the Royal Dutch Library (Koninklijke Bibliotheek): https://www.kb.nl/en/research-find/for-researchers/data-services-apis-and-downloads
Persée
Persée allows free and open consultation and use of complete collections of scientific publications. Created by the Ministry of National Education, Higher Education, and Research, it was put online in 2005.

more... less...

API not available.
Sacred Texts
Online archive of freely-available books about religion, mythology, and folklore.

more... less...

Downloading: Files available as HTML. Most of the books are in the public domain, but not all of them; be sure to check the license for the book(s) that you are using. (For more info, see here: https://sacred-texts.com/cnote.htm)
Early English Books Online (EEBO)
Collection of digitized English printed works from the period 1470-1700. A significant number of these works contain full text transcriptions.

more... less...

Downloading: multiple documents can be downloaded at once in different formats.

Nexis Uni
Newspaper archive with full articles from leading newspapers in the Netherlands. These include: De Volkskrant, Het Parool, De Telegraaf, Trouw, Algemeen Dagblad, and NRC Handelsblad. Dutch and international news with full articles from hundreds of local, national, and international newspapers, magazines, news agencies, publications and international news from English and non-English sources. There is also access to Dutch and international company information.

more... less...

Downloading: bulk downloads possible of up to 500 documents at a time (PDF, Word or RTF); API not available within our license.
Reddit
Reddit is an online network of communities covering a wide range of topics, where users can create and share content and have longer-form discussions. Most users are anonymous.

more... less...

Access: Reddit4Researchers, a community that will enable and regulate researcher access to Reddit data, is currently in beta form, with bulk downloads for a small group of users via API with an SQL or Python wrapper. For more information, see www.reddit.com/r/reddit4researchers.
X (Twitter)
X is a social media website where users share content, typically in a short-form (e.g., via short text messages, a.k.a. "tweets"). Content tends is largely concerned with current events. Users can be anonymous, but most are not.

more... less...

Access: For more information, see https://developer.x.com/en/use-cases/do-research/academic-research.

arXiv.org
arXiv.org is a repository of electronic preprints of scientific papers in the fields of mathematics, physics, astronomy, computer science, quantitative biology, statistics, and quantitative finance. Produced by Cornell University.

more... less...

Downloading: bulk downloading is possible through an API. Terms of use: https://info.arxiv.org/help/api/tou.html
BioMed Central
Contains articles from approx. 300 open access journals in science, technology, engineering and medicine. Part of Springer Nature.

more... less...

Downloading: API available: https://dev.springernature.com/#/Openacces
TDM information: https://www.springernature.com/gp/researchers/text-and-data-mining
CORE: Open Access Research Papers
CORE provides a central API to access full content from tens of thousands of openly available scientific publications from thousands of OA repositories. They also provide full datasets by request.
Corpus Thomisticum
Corpus Thomisticum provides almost all Thomas Aquinas' works in Latin edition. In addition to this, it contains scans of manuscripts, a bibliography of publications on Thomas Aquinas, and several research tools.

more... less...

Downloading: the text is offered as flat-text on the website.
EUR-Lex
EUR-Lex gives access to EU legal documents and case-law.

more... less...

Download/file type: PDF, CSV, Excel, XML, bulk download.
API available.
https://eur-lex.europa.eu/content/help/data-reuse/reuse-contents-eurlex-details.html
LSEG Workspace (Datastream)
Central hub for broad and deep information – from datasets and analytics to trusted news and content. Regarding text mining, this database is especially useful for gathering Annual Reports and other SEC filings.

more... less...

The software is freely available to download and accounts can be requested with the NSM Data Team: datasupport-nsm@ubn.ru.nl
API available: Please contact the NSM Data Team for more information.
Nietzsche Source
Nietzsche Source is a web site devoted to the publication of scholarly content on the work and life of Friedrich Nietzsche.
Contains the Digitale Kritische Gesamtausgabe.

more... less...

Downloading: the text is offered as flat-text on the website.
OECD iLibrary
OECD iLibrary contains thousands of e-books, chapters, tables and graphs, papers, articles, summaries, indicators, databases and now also Podcasts - discoverable by theme, country or content type.

more... less...

API available: https://data.oecd.org/api/
Officiële bekendmakingen
Part of overheid.nl, in particular for official statements.

more... less...

API available: https://data.overheid.nl/ondersteuning/data-publiceren/api
Overheid.nl
This website is a central access point to all information about government organisations of the Netherlands.

more... less...

Download/file type: PDF.
API available: https://data.overheid.nl/ondersteuning/data-publiceren/api
Past Masters
Editions and translations of the works of a great number of classical philosophers.

more... less...

Downloading: the text is offered as flat-text on the website.
PLOS
Public Library of Science is an Open Access publisher that provides free access to peer-reviewed articles and preprints.

more... less...

Description: Public Library of Science is an Open Access publisher that provides free access to peer-reviewed articles and preprints.
Access: ✔.
TDM information: https://plos.org/text-and-data-mining/
Download/file type: zip file contains JATS-standard XML content of articles without figures and supplemental data. Also PLoS API: https://api.plos.org/text-and-data-mining.html
PMC Open Access Subset
PubMed Central is a full-text open access archive of biomedical journal articles provided by the National Library of Medicine (NIH/NLM)

more... less...

Description: PubMed Central is a full-text open access archive of biomedical journal articles provided by the National Library of Medicine (NIH/NLM)
Access: not all articles in PMC are available for text mining. Copyright statements per article.
Download/file type: XML, PDF, plain text, bulk download possible under specific conditions
TDM information: https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/
Several API's available: https://www.ncbi.nlm.nih.gov/pmc/tools/developers/
The Post-Reformation Digital Library
A database that links to openly-available literature on theology and philosophy between the late 15th-18th centuries, from a variety of online sources.

more... less...

Downloading: Download options and licenses depend on the source.
Rechtspraak.nl
The official website of the Dutch judiciary that contains information on the judicial system and a selection of case law.

more... less...

More information Open Data, see here: https://rechtspraak.nl/Uitspraken/Paignas/Open-Data.
United Nations Documents
On this website, you can search for meeting records, summaries of conferences, UN reports, treaties, maps, yearbooks and more.

more... less...

API available: https://data.un.org/Host.aspx?Content=API
Westlaw
Database with full-text case law, legislation, treaties, law journals, journal articles, books and news concerning the United States, Australia, Canada, the European Union, Korea, Hong Kong and the United Kingdom.
Westlaw UK