Text Data Preparation with R
Text-mining as the means of extracting and analysing text data with the use of computer software or packages from a corpus of textual sources such as documents, correspondences, books and journals into understandable and meaningful patterns and relationships hidden in the text is becoming popular in digital scholarship research. This technique can be understood as the application of linguistic, statistical and machine learning methods to a set of structured or unstructured textual sources in order to make sense of the content with minimal manual effort. Many of the computer software or packages for this task are developed to analyse English language materials. It is quite demanding for computers to process and analyse natural language. It is even more difficult to handle Chinese language text in ancient writing style.
The CUHK Library collects a lot of rare and precious Chinese language materials and has turned them into digital forms. The Library’s Digital Scholarship team is interested to experiment with text mining technique to shed light on the Library’s Chinese digital collections with this project. It aims to make use of “R” to process and analyse Sheng Xuanhuai Collection owned by the Art Museum of CUHK to demonstrate how computational text processing and analysis can be done for Chinese texts.
In this project R segmenting packages (e.g. jiebaR) was selected and applied to two volumes initially to demonstrate the proof of concept. The entire process of the computational text processing from setting up the R environment to the creation of a matrix of word counts (term-document matrix (TDM)) and the wordcloud is described in this project. All the computer codes are available here. This is only the first stage of work. It is hoped that its detailed documentation will enable researchers to further work on other Sheng documents to bring to the possibilities of text analysis such as text clustering or topic modelling as for the next stage of development. Users are welcome to read the report for further information on the process and download the code for reuse.
Project creator: Dr. Yun Tai, former Post-doctoral Fellow, Digital Scholarship Team
Enquiries: dslab@lib.cuhk.edu.hk