Processing ECB Data with Common Lisp

Description of the problem

European Central Bank has a large economic data repository and lets people use it for free. It even has a nice API. I sometimes use their datasets as examples when I teach, especially when it comes to time series stuff.

Today, I am going to show how one can use their API from common lisp which is not very complicated, and one can translate the code to other languages in a fairly straight-forward manner.

The hard part

The data repository contains a large number of datasets. The good thing is that ECB allows bulk downloads of data. There is also a separate interface for data exploration, but that’s a topic for another post. Today, I am going to assume you know which dataset you want from ECB. Use their data exploration interface to find something that catches your fancy for the examples below.

The easy part

First, let us load all the libraries we need:

;; (mapcar #'ql:quickload '(:drakma :xmls :flexi-streams))
(mapcar #'require '(:drakma :xmls :flexi-streams))

(NIL NIL NIL)

Next, the threading macro from clojure that I like using. It is going to make the code more readable.

(defmacro ->> (x &rest forms)
  (dolist (f forms x)
    (if (listp f)
        (setf x (append f (list x)))
        (setf x (list f x)))))

->>

Here is another function that I like and going to use for this post from clojure:

(defun juxt (&rest fns)
  (lambda (x)
    (mapcar (lambda (f) (funcall f x)) fns)))

JUXT

Next, I need a piece of code that handles the requests for the ECB API.

(defun request (dataset)
  (progn
     (setf (aref dataset 3) #\/)
     (->> dataset
          (concatenate 'string "https://sdw-wsrest.ecb.europa.eu/service/data/ECB,")
          drakma:http-request
          flexi-streams:octets-to-string
          xmls:parse)))

REQUEST

And the piece of code that extract the information as an association list:

(defun deep-get (tags xs)
  (if (or (null xs) (null tags))
      xs
      (deep-get (cdr tags)
                (let ((tag (car tags)))
                  (mapcan (lambda (x) (xmls:xmlrep-find-child-tags tag x)) xs)))))

(defun extract-data (dataset)
  (->> dataset
       request 
       list
       (deep-get '("DataSet" "Series" "Obs"))
       (mapcar (apply #'juxt
                         (mapcar (lambda (tag)
                                 (lambda (x) (->> (xmls:xmlrep-find-child-tag tag x)
                                                  xmls:node-attrs
                                                  cadar)))
                     '("ObsDimension" "ObsValue"))))
       (mapcar (lambda (x) (cons (car x) 
                                 (read-from-string (cadr x)))))))

DEEP-GET
EXTRACT-DATA

Let us test: I am going to use data on Europe’s total trade (exports and imports) with Turkey. The keys for those datasets at ECB API are “TRD.M.I8.Y.X.TTT.TR.4.VAL” and “TRD.M.I8.Y.M.TTT.TR.4.VAL”

(defparameter turkey-export (extract-data "TRD.M.I8.Y.X.TTT.TR.4.VAL"))
(defparameter turkey-import (extract-data "TRD.M.I8.Y.M.TTT.TR.4.VAL"))

TURKEY-EXPORT
TURKEY-IMPORT

which then can be plotted