Author: rodeo

Posted on

History of Deep Learning

This paper presents a history of deep learning from Aristotle to the present time. The different milestones are summarized in this table:

So it is clear that many ideas date to several decades ago. In the article, the authors conclude:

This paper could serve two goals: 1) First, it documents the major milestones in the science history that have impacted the current development of deep learning. These milestones are not limited to the development in computer science fields. 2) More importantly, by revisiting the evolutionary path of the major milestone, this paper should be able to suggest the readers that how these remarkable works are developed among thousands of other contemporaneous publications. Here we briefly summarize three directions that many of these milestones pursue:

  • Occam’s razor: While it seems that part of the society tends to favor more complex models by layering up one architecture onto another and hoping backpropagation can find the optimal parameters, history says that masterminds tend to think simple: Dropout is widely recognized not only because of its performance, but more because of its simplicity in implementation and intuitive (tentative) reasoning. From Hopfield Network to Restricted Boltzmann Machine, models are simplified along the iterations until when RBM is ready to be piled-u
  • Be ambitious: If a model is proposed with substantially more parameters than contemporaneous ones, it must solve a problem that no others can solve nicely to be remarkable. LSTM is much more complex than traditional RNN, but it bypasses the vanishing gradient problem nicely. Deep Belief Network is famous not due to the fact the they are the first one to come up with the idea of putting one RBM onto another, but due to that they come up an algorithm that allow deep architectures to be trained effectively.
  • Widely read: Many models are inspired by domain knowledge outside of machine learning or statistics field. Human visual cortex has greatly inspired the development of convolutional neural networks. Even the recent popular Residual Networks can find corresponding mechanism in human visual cortex. Generative Adversarial Network can also find some connection with game theory, which was developed fifty years ago.

Coming from the field of economics and game theory, we cannot agree more especially when we read the literature of reinforcement learning (RL) or generative adversarial network. Once we talk about strategic agents with objectives and payoffs to maximize it is very similar to economics. There are differences between economics and machine learning in the approach of solving these problems and we will discuss some research that studies them.

Posted on

Why Should I Trust You?

A challenge of complex machine learning models is to develop trust in the models. If it is a black box some users might not be feel comfortable using them. Models need to be interpretable, meaning that users should be able to understand how the outputs (predictions) are generated from the inputs (features).

Different approaches have been suggested. A recent one is a technique called Local Interpretable Model-agnostic Explanations (LIME). LIME approximates a model with an interpretable model locally. An interpretable model is a model such as linear models with a limited number of features.

A short video introduces the approach.

You can read the paper here.

 

Posted on

Gradient Boosting Machine Learning

Machine learning has a long list of methods to learn from data. Among them is gradient boosting machine learning as taught here by Professor Trevor Hastie from Stanford University.  In this video, he introduces and compares decision trees, bagging, random forests and boosting.

He has authored an excellent book, The Elements of Statistical Learning than you can download here.

Posted on

AI Code

The UK House of Lords recently published a report on “the economic, ethical and social implications of advances in artificial intelligence.” It suggested an AI Code to reassure the public that AI will not undermine it. The principles are:

(1) Artificial intelligence should be developed for the common good and benefit of humanity.

(2) Artificial intelligence should operate on principles of intelligibility and fairness.

(3) Artificial intelligence should not be used to diminish the data rights or privacy of individuals, families or communities.

(4) All citizens have the right to be educated to enable them to flourish mentally, emotionally and economically alongside artificial intelligence.

(5) The autonomous power to hurt, destroy or deceive human beings should never be vested in artificial intelligence.

This reminds us of course of Asimov’s Three Laws of Robotics:

(1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.

(2) A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

(3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.

We believe that it is only the beginning of our reflections on how to regulate AI. There is already some work on legal liabilities of AI. You can read this interesting paper on Artificial Intelligence and Legal Liability.

Posted on

Computing Machinery and Intelligence

One of the most seminal papers written on artificial intelligence was by Alan Turing in 1950. The paper describes the famous Turing test to determine if machines can think. We encourage you to read it.

Alan Turing calls it the Imitation Game. It involves three parties A, B and C. A is the machine, B is a human and C interacts with both A and B by text to figure out if A is a human or a machine and B is helping C. If the probability of success of C does not change whether A is a human or machine then Turing suggests that “machines can think”.

This indirect approach has the advantage of being more objective that directly addressing the question of whether a machine can think. The disadvantage is that it does not differentiate between pretending to think and thinking. A very good imitation could win the imitation game!

It was also mentioned in the movie The Imitation Game:

https://www.youtube.com/watch?v=IwVzwsam1NM

We feel that the question has now be answered at least in some specific  domains such as games (see AlphaGo). It is now demonstrated that computers can be better than humans. It would be strange to still argue that computers do not think when hard thinking humans cannot beat them in such intellectual tasks.

Some confusion arises when thinking and consciousness are deemed to be equivalent. Turing cites the objection of a Professor Jefferson:

[The Argument from Consciousness] This argument is very, well expressed in Professor Jefferson’s Lister Oration for 1949, from which I quote. “Not until a machine can write a sonnet or compose a concerto because of thoughts and emotions felt, and not by the chance fall of symbols, could we agree that machine equals brain-that is, not only write it but know that it had written it.
No mechanism could feel (and not merely artificially signal, an easy contrivance) pleasure at its successes, grief when its valves fuse, be warmed by flattery, be made miserable by its mistakes, be charmed by sex, be angry or depressed when it cannot get what it wants.”

If we consider birds it is clear that they are thinking creatures. We do not know if they are conscious though some researchers believe that consciousness is not restricted to humans (see The Cambridge Declaration of Consciousness).

Now if we compare AlphaGo to a bird, it is easier to conclude that AlphaGo is thinking as much as a bird and is even “smarter” than a bird in many domains. We do not need to investigate if AlphaGo has a conscience.

On the last point we note that some recent research of the Theory of the Mind seems to give a machine the ability to represent mental states of others including their desires, beliefs and intentions. It might be possible for the machine to apply the same model to itself. This will make the machine close to being conscious. This will be subject of another post.

Posted on

Welcome to Rodeo AI

Rodeo AI is an economics blog on data science, machine learning and artificial intelligence (AI). We plan to explore the exciting field of AI, its recent developments, its applications and its consequences from an economist’s perspective. We want to explore how these new fields can help improve the economist’s work but also how economics can be useful to the data scientist, machine learning and AI specialist.