Turkish Sentiment Analysis Using Thesaurus Distance

Description of the problem

In my last post I implemented a word distance algorithm using a thesaurus for English. Today, I will do the same for Turkish. It is difficult to find a free Turkish Thesaurus. After some digging around I found one. The Turkish Thesarus developed by volunteers and maintained by Anil Ozbek.

Implementation

Let us look at the structure of the thesaurus file:

aba|2
(noun)|abla|anne
(noun)|barak|çuha|çul|keçe|kebe|şayak|palas

In the file “|” symbol is used to separate fields. The first field is the word, the second is the number of distinct senses of the word listed here. In this example the word is “aba” and it has 2 senses listed here. Then the next two lines are these senses. The first field designates which part of speech the sense belongs to followed by each synonym separated by “|”.

Here is the lisp code to load the thesaurus

(defvar data 
   (let ((res (make-hash-table :test 'equal))
         (vertex ""))
      (with-open-file (infil "../data/turkish_thesaurus.txt" :direction :input)
         (read-line infil nil)
         (do ((line (read-line infil nil) 
              (read-line infil nil)))
             ((null line) res)
             (if (char= (elt line 0) #$)
                 (dolist (word (cdr (ppcre:split #\| line)))
                    (push word (gethash vertex res))
                    (push vertex (gethash word res)))
                 (setf vertex (car (ppcre:split #\| line))))))
         (maphash 
            (lambda (x y)
                (setf (gethash x res) (remove-duplicates y :test 'equal)))
            res)
         res))

DATA

I will use the same distance function as before

(defun dist (x y &optional (n 5))
   (let ((res (gethash x data))
         (i 0)
         (temp nil))
      (loop while (and (not (member y res :test 'string-equal)) (< i n)) do
         (incf i)
         (setf temp (loop for j in res append (gethash j data)))
         (setf res temp))
      i))

DIST

and a simple test:

(dist "hastalık" "korku")

1

(dist "baba" "gurur")

3