- Good afternoon everyone.
It's a great pleasure for me to introduce professor
Ariel Procaccia from Carnegie Mellon University.
Ariel does amazing work at the intersection of
artificial intelligence, theory, and economics.
His work spans many, many different topics,
ranging from fair division, to voting,
to machine learning and beyond.
He does foundational work on the theoretical side,
and he takes his ideas all the way to practice.
For example, building some non-for-profit website called
spliddit.org that you can take your fair division
problems to and a number of other things of that nature.
He's recipient of the IJCAI Computers and Thought Award,
Sloan Research Fellowship, NSF Career,
and a number of other awards, many best paper awards.
And it's a great pleasure for us
to have him talk about democracy.
- Thank you very much Anna.
It's a pleasure to be here.
So the topic of the talk is extreme democracy.
And I want to start with this quote from Pia Mancini.
She is an Argentinian politician and activist.
She gave a very popular Ted talk of couple years ago,
where she quoted Marshall Mcluhan who was a
Canadian professor, saying that politics is solving
today's problems with yesterday's tools.
Now you may be wondering why
I'm not quoting Marshall Mcluhan directly.
The reason is that he didn't actually say that.
He said that "Our 'Age of Anxiety' is the result
"of trying to do today's jobs with yesterday's tools."
Nevertheless, I really like Pia Mancini's version
of the quote about politics.
So I'm going to, you know, attribute this quote to her.
And the point I want to make here is closely related
to this, which is that the world has really changed
in the last couple of decades because of modern technology,
but the way in which we do democracy
hasn't fundamentally changed for centuries.
And it seems like a really good time to think
about, you know, how we do democracy.
Think, you know, rethink democracy to some degree.
And the theme of the talk is going to be that
computer science can play a big role in this discussion,
both in terms of thinking about the practice of democracy,
how we do democracy and also in terms of thinking
about expanding the potential reach of democracy,
so thinking about new application domains where this
combination of computational thinking
and democratic thinking, if you will, can give us
new ways to accomplish our goals.
So I'm going to illustrate this theme through three topics
that we've worked on in the last year or so.
The first is liquid democracy which is a new paradigm
that allows voters to delegate their votes to others.
The second is participatory budgeting which is the
idea that a local government, usually a city,
engages its residents in the process of allocating
the budget or some part of the budget.
And the last part will be virtual democracy which
is a term I came up with for the purposes of this talk.
But the idea is that we are going to have a virtual
election among machine learning models of people,
and we're going to aggregate their virtual
preferences to make a decision.
And I'll show you how we're thinking about applying
this kind of approach in order to automate
decisions on ethical dilemmas.
So let's get started with liquid democracy.
This is joint work with Anson Kahng and Simon Mackenzie,
and I'll give you the full references
at the end of the talk.
So a good place to start here is just positioning
liquid democracy in the context of other forms
government that you're all familiar with.
So one is monarchy or dictatorship, something
that's been popular for many years in history.
So here we have a king or a queen or a dictator,
and when it's time to make a decision, there's
only one vote that counts and that's the
vote of the king or queen.
On the other extreme, we have what's called
direct democracy which was famously
practiced in ancient Athens.
And the way it worked there is everybody would show up.
All of the citizens would show up to the same place.
And everybody would vote directly on the issue at hand.
Now one caveat is that of course not everybody
could be a citizen.
Women were not citizens.
Slaves were not citizens.
But overall this allowed
for some measure of wide participation.
A big shortcoming for this kind of approach
is that it doesn't scale.
So if you think about today's world with all of its
complexities, it seems unrealistic to expect
every citizen to be informed about every issue
and really have the time to express an opinion
about, you know, all of the issues that come up.
So partly for that reason, we have what's called
representative democracy which is, you know...
The way it works is we have professional politicians
whose job it is to be aware of the various issues.
We elect some of them to Parliament,
and when it's time to vote, there the ones who
actually vote on our behalf.
So this works reasonably well in many cases.
The, you know, one shortcoming of this approach is
that our representatives don't always vote according
to the way we would want them to vote.
And even worse, they may not vote with our
best interests at heart.
So this motivates, this new paradigm, liquid democracy
that you can think of as lying between direct
democracy and representative democracy.
So the way this works is every voter can vote directly
on the issue at hand, or they can choose
to delegate their vote to another voter.
And the crucial point is that
these delegations are transitive.
So if I delegate to Anna and Anna delegates her vote
to Pedro, Pedro would vote on behalf of both Anna and me.
So here's an example where we have a delegation graph,
a directed edge from i to j says that i delegates to j.
So what would happen here is that when we're actually
voting on some issue, the sinks of this graph,
the ones that don't have outgoing edges were
not delegating or the ones who actually vote.
So in this situation we're going to have two voters,
this one and this one, who only vote on their own behalf.
We have one voter here with the weight of two that
votes on behalf of one other voter.
And we have this super voter with the weight of six
that votes on behalf of five other voters, everybody
with a directed path to this voter.
So to me this seems like a really compelling idea,
and its been really gaining steam in the last few years.
So there's been a number of systems that have gained
some traction built on liquid democracy.
One is the liquid feedback system which is, for example,
being used by the German Pirate Party.
It's a funny sounding party, but it actually had quite
a bit of political power a couple of years ago.
And they used the system, for example, to collaboratively
decide on the party platform, what they want to advocate.
More recently there have been a couple of parties
that basically suggested to hack the political system
by running for Parliament as usual.
But what they said was if we get elected to Parliament,
our representatives would vote according
to the wishes of the party members,
and those wishes will be determined
by a liquid democracy based system.
So one example is the Net Party in Argentina.
Pia Mancini is actually one of the founders of this party.
They have a system called DemocracyOS, which is
built on liquid democracy.
Another example is the Flux Party in Australia.
They have a system also called Flux.
Now unfortunately, neither party actually won
any seats in Parliament.
But this was like a nice attempt that did attract
quite a bit of attention
to the idea of liquid democracy and its potential.
So what I want to do is think about liquid democracy
in a formal model and try to reason a bit
about, you know, when it works well
and especially when it doesn't work well.
So in this model, we're going to have a directed
graph G, where the vertices coincide with voters,
and we have an edge from i to j if voter i knows voter j
or even knows of voter j.
Now I'm going to assume that
there are only two alternatives.
So it's a binary issue that we're deciding on.
And moreover one of them is correct and one is incorrect.
So I'm thinking about a case where we have some ground truth
and what voters are trying
to do is determine this ground truth.
So for example, think about a policy decision like
tax reform, and imagine that we're evaluating it
according to some objective measure, like will, you know,
by how much will the national debt increase in five years.
So according to this objective measure, one alternative is
better than the other.
Right, under one the national debt will increase less
than under the other, maybe even would go down.
So think of this better alternative according
to the objective measure as the correct alternative,
and we think of the other alternative
as the incorrect alternative.
Now when there are only two alternatives,
there's only one reasonable to make,
way to make decisions, which is majority vote.
And remember that under liquid democracy,
we could have some delegations.
So we're going to look at the majority of weight
given to any of the two choices.
Now in addition, every voter has a confidence level pi,
which is his probability of voting correctly.
So the way to think about this is when it's time
for me to vote, I try to estimate, what's, if I did
decide to vote, I try to estimate what's the correct
decision according to the information available to me
and whatever knowledge I have.
And then, you know, I try to make a determination,
and I'm going to get it right with probability pi,
and I'm going to get it wrong
with probability one minus pi.
Now another important definition is
this idea of approval.
So we say that voter i approves j if i knows j,
if the edge ij is in the graph.
And moreover crucially, voter j is more competent than i
by a margin of at least alpha or more than alpha
for some parameter alpha of the model.
So this is going to be a necessary condition
for delegation, the fact that i knows j
and also we have this margin incompetence.
This is of course a very strong assumption, right,
because it says that we can only delegate
to people who are more competent.
So this really helps liquid democracy.
And the way I want to justify it is by noting that the
main result that I want to tell you about is a negative
result against liquid democracy.
So by making this assumption that really helps liquid
democracy, I'm only going to make
the negative result more powerful.
So let's see an example of where things might go wrong.
So this is our graph G.
It's a star.
It has only eight arms in the illustration,
but think about many arms.
And the competence level of the arms will be 0.6.
And the competence level of the center is 0.8.
So it's significantly more competent.
Let's say that alpha, this margin is less than 0.2.
So under direct democracy, all of the voters would vote
independently, and then the expected number, fraction
of correct votes will be close to 0.6 times the
number of vertices.
And with high probability, the actual number will
be close to the expectation.
So with high probability,
we will have a majority of correct votes.
Alright so, you know, this follows
from any concentration equality.
It actually dates back. - [Male Student] Independent.
- Independent, yeah, so the voting is
going to be independent.
- [Male Student] People's private
information is independent.
- Yeah, so there's no correlation whatsoever.
Yeah, notice by the way, you know, this example
can be modified even if we do have some correlation.
Let me say a few words about this in thirty seconds.
So, you know, under direct democracy we will
have the right decision with high probability.
Under liquid democracy, you know, in an eve delegation
model, everybody would delegate to the center.
The center is more competent, so the center gets
it right with probability 0.8.
But the center also makes a mistake
with probability 0.2 which is pretty significant.
So here even though everybody delegated to a more
competent vertex, we're actually doing worse than before.
And just to note, you know, regarding correlation,
so for example one thing you could say here is maybe,
you know, everybody depends on the center.
So if the center is correct, the arms will vote
correctly with probability 0.61.
And if the center is incorrect, it would vote, vote
correctly with probability 0.56.
So it would still give you 0.6 probability of being correct.
So of course, like the example would be robust
to this kind of thing.
Right so what you would get, I guess, is the arms
are conditioning, dependent of each other,
given the center in this example.
And things would still work out.
And some of the stuff I'll talk about can be
extended to allow some measure of correlation,
but we will assume independence.
- I think what, naturally with the case is that it's
the most highly correlated voters
that choose the same representatives.
So it's exactly the opposite of the situation.
It's people who are agree you're like, who'll
tend to pick the same representative.
- Right, but here. - Which makes this less bad.
- Yeah, I mean in this, obviously, right,
so I guess in this social network we have the
situation where people are not really connected
to each other.
Everybody's connected to the center.
Right, so here the only reasonable model of correlation
would be with the center.
But you're absolutely right, like, if, if we did
assume such a model, that would only strengthen
this kind of result.
Okay, so what we want to do is think about whether
we can get around this kind of phenomenon,
if we allow for more clever delegation mechanisms.
So what's a delegation mechanism?
It's going to be a function that observes
the underlying graph G and the approval relations.
So it can't observe the exact competence levels,
but it does have access to the approval relation.
And it decides possibly at random, whether each voter votes
or delegates and to whom.
So I want to specifically think about local delegation
mechanisms, which are mechanisms where the decision
for voter i is based only
on the approved neighbors of voter i.
So it would look at the local neighborhood of i,
and make a decision based on which subset of the neighbors
are actually approved.
The motivation for this is that this is closer
to what we have in the real world today,
where voters make their own
delegation decisions independently,
only based on their local information.
So this would actually give the mechanism a bit more power
to make delegation decisions.
So examples of these kinds of mechanisms are delegate
to a uniformly random approved neighbor,
you know, don't delegate at all with probability 1/2
and delegate to a specific
approved neighbor with probability 1/2,
or even don't delegate at all.
Like if it says nobody ever delegates, that would just
coincide with direct democracy.
So what are the properties that we want
with a fine tune minimal properties that we
would want delegation mechanisms to satisfy?
The first is basically formalizing the property that
we had on the previous slide.
So we call it do no harm.
And slightly more formally, what it says is that
for any epsilon, the loss, which is the difference
between the probability of liquid, of direct democracy
making the right decision and the probability of liquid
democracy making the right decision, is at most
epsilon on all sufficiently large instances.
Or in other words, the loss would go to zero
as the instances grow larger.
Now this can be achieved if you never delegate.
If you just do direct democracy, because direct democracy
wouldn't lose to itself.
So to make things slightly less trivial, we introduce
another minimal property that's called positive gain
which just says that there are some examples
where liquid democracy actually
outperforms direct democracy.
So we can just do direct democracy.
And the negative result that I advertised before,
you know, basically says, that for any value of alpha,
the parameter that governs when we delegate,
the gap needed for delegation, there is no local
delegation mechanism that satisfies the do no
harm and the positive gain properties.
So there's no local mechanism that would satisfy
these two basic properties.
So the proof of this theorem has some details that requires
some calculations, but I think that the intuition
behind it is very simple.
So I want to try to illustrate the intuition to you
in a couple of minutes.
So the basic idea is that if you have a mechanism
satisfying, yeah.
- Looking at different instances, like the ability of
individual voters to get each of them right won't change?
- [Lecturer] I misheard the question.
- In a democracy, like, one of the problems was that
there too many things to decide about, and each individual
doesn't have the ability to think about all of them.
That means that when it comes to different things,
I have different ability to make the connections.
You're like the abilities should remain the same or?
- Right, so the question was, you know, basically
whether the competence levels
are different for different issues.
So all of this is done from the viewpoint of a single issue.
We're thinking about, we're fixing the issue,
and we're saying the competence levels are specific
to this issue.
There's another aspect, I think, of your question
which is, you know, why are we all measuring this
issue according to the same criterion.
Right, when we think about competence level, it's the idea
that we would make the right decision according
to a fixed criterion.
For that this is another assumption, but it's again
an assumption that helps look at democracy,
because we're all thinking about optimizing the same thing.
So it's again an assumption that only strengthens
this negative result.
Okay, so what's intuition here?
If we have the positive gain property, there must be
some neighborhood where delegation actually happens.
Right otherwise we would just be doing direct
democracy and would never be able to gain on any example.
So let's say that this is such a neighborhood.
And we're taking the viewpoint of the gray vertex.
Right, and here what you saying is the gray vertex has
three approved neighbors that are marked by dashed edges.
And one neighbor in the underlying
graph that is not approved...
Sorry, the solid edges are the approved neighbors.
The dashed edge is the neighbor in the underlying graph
which is not approved.
So what I'll do is I'll set competence levels
for the vertices that will
realize this approval relationship.
So we're going to have three high competence vertices
and two low competence vertices, including the one
whose viewpoint I'm taking.
And next, what I'll do is I'm going to take this
neighborhood and replicate it in a way that keeps the
same blue vertices, but replicates the green vertices.
So we got something like this.
And the crucial point is that from the viewpoint of each
one of the green vertices, the neighborhood looks exactly
the same as what we initially had.
So a local delegation mechanism would, you know,
have every green vertex delegate to the blue vertices
with some constant probability,
because that's where we started from.
Now in addition we're going to add the cloud of these
medium competence vertices that are disconnected
from everything else.
Setting their competence level is kind of tricky,
so it requires a bit of work
to even show that what we want is feasible.
But the way we do it is so that if everybody votes
independently under direct democracy, then getting
something close to the expected fraction of green
vertices to vote correctly is enough for the majority
vote to be correct.
On the other hand, the other liquid democracy,
like we said some fraction of the green vertices
delegate to the blue vertices.
Now without that fraction, the majority will be wrong
with high probability.
And in addition, now all of these blue vertices
have a lot of weight, and there's only a constant
number of them.
So with some constant probability
all of the vote incorrectly, and in that case the majority
decision is incorrect.
So what we get is that under liquid democracy, we have
a constant probability of making
an incorrect majority decision.
Under liquid democracy we have a correct decision
with high probability, and that creates this gap
that violates the do no harm property.
So what we see from this construction is that local
delegation mechanisms are inherently flawed,
because they can't identify situations where few
vertices amass a large number of votes.
Of course, with no local mechanisms,
we can get around this problem.
So one, you know, naive idea would be to define this
greedy cap mechanism that just says, you know,
let's greedily delegate votes according to what voters want,
but we'll set a cap on the weight that can be amassed
by any single voter, some caps given.
And then, you know, if you've already reached this cap,
we say you can't get any more delegations.
- So in the situation with representative democracy,
you really have, essentially, a situation, you have
this constant probability of having a bad situation,
because we only have a constant number of representatives.
And so, in some sense, it's a local mechanism where
those blue vertices are representatives
or potential representatives, really.
- Yeah, I agree.
So Paul's point was, you know, this kind of situation
can happen in representative democracy as well,
because we're correlating all of the weight
on our members of Parliament,
and there aren't that many of them.
And I agree.
So essentially with liquid democracy, the question,
or what we wanted to achieve, was designed delegation
mechanisms in a way that achieved, that circumvents
this kind of problem.
Right, so with representative democracy, we're stuck
in a pretty rigid system.
Here we have more flexibility in how we think
about when people delegate to others, people can vote
directly or delegate.
And it, you know, we were hoping that even with local
mechanisms, we could get around this kind of problem.
What the theorem tells us is that we can't actually
with local mechanisms.
Right, but however with non-local mechanisms, we
can get around this problem.
In particular, notice that we will avoid the situation
where that we have in representative democracy,
with a lot of weight on particular representatives.
And what we show is that under, another truly mild
assumption, if we set the cap correctly, then the
greedy cap mechanism would satisfy both the positive
gain and due no harm properties.
So the proof of this is very long.
I think there must be an easier way to prove it than we did.
And I don't want to go into the proof.
I do want to talk a bit
about the conceptual implications of this result.
So one interesting point is that people who have thought,
who have been thinking about liquid democracy, have
been really worried about this phenomenon where few
voters have amassed a large weight.
In particular in the German Pirate Party, this guy
was the superstar.
So he is a linguistics professor in Germany,
and he amassed so much weight that according to this
article in Der Spiegel, quote, "His vote was like a decree."
So basically, when he made a decision he would shift the
majority in the direction that he wanted.
And of course, you know, many people who are not this guy,
were unhappy about the situation
and thought that it's not democratic.
So of course greedy cap would get around this situation
by imposing this cap and preventing people
from amassing massive weight.
But what I want you to take from this part of the talk
is not that greedy cap is a great mechanism,
because that's not what I'm saying at all.
Remember that we only designed it to achieve two very
basic properties that are not very compelling
as a positive result.
And moreover we made assumptions
that really helped liquid democracy.
So what I want you to take away is actually the negative
result, the fact that local delegation mechanisms are
inherently flawed and that centralized delegation
decisions could be the next step
for effective liquid democracy systems.
In our ongoing work, we've been thinking
about, you know, how to design centralized delegation
mechanisms from an algorithmic perspective.
And that's something I'll be happy to talk about offline.
Yeah?
- The fact that the edges in the underlying graph are
zero one, they either exist or they don't seems like a
dramatically simplifying assumption, you know.
I know you or I don't.
In particular relevant to centralized delegations.
Do you have thoughts on this?
- Yeah, I mean, I don't view it as a dramatic,
that particular assumption as being dramatic.
So it, essentially what we're saying is I can only,
you know, I can only delegate to j if I know j.
So he knows of the existence of j and like could
potentially delegate to...
- If I think of it as I'm allowed to delegate.
- Exactly, exactly so I just think of it as, like,
part of the necessary condition for delegation together
with the, this gap in competence, right.
Yeah, Pedro.
- I think the basic motivation for liquid democracy isn't
necessarily that the representatives will make better
decisions is that people just don't have time.
So there seems like there's a very important thing
missing here which is like, yes, you know, liquid
democracy will be worse than direct democracy,
but we would have to offset against the gain that,
you know, people that way have looked.
Otherwise they might not even vote.
- Right, so the way I think about it is, you know,
people not having time translates to a low competence level.
If you have made them make, you know, make a choice
then they wouldn't be well informed
and would make a right choice with low probability.
And for that reason, you know, when we, when we delegate,
like we would expect things to be better.
At least like initially, we were hoping for things
to be better under liquid democracy.
So I think, you know, ideally liquid democracy
should get around this problem of low competence
by allowing delegation to more competent people
and that's what this negative result is trying to capture.
- For the sake of the role and point of comparison,
or current representative democracy
and not direct democracy.
- Right, so as Paul was pointing out, I mean the
question was about comparison to current representative
democracy, as Paul was pointing out, the kind of issue I was
talking about, when you think about it in this mathematical
model is also an issue in current representative democracy,
where we're focusing a lot of weight on this like
small number of representatives
that could make wrong decisions.
- What I mean though is that, as far as the theoretical
results, it might be worse than direct democracy
but better than representative democracy.
- Oh I see, right.
So that's a good point.
Well I guess you would need like a completely different
model for, you know, if you wanted to think
about, you know, instead of just saying we have these
representatives and the graph structure is like this,
trying to model what representative democracy looks like
and when it's likely to be successful.
This model wouldn't give you that.
The reason why we're comparing to direct democracy
is because of the feeling that this is essentially
in some sense the ideal weight, where everybody's
participating, and this is, and this is what we
would have been doing if we didn't have this problem
of low competence levels.
Right, so, so the motivation is coming from the fact
that we would want direct democracy.
It doesn't scale because of low competence levels.
Therefore we allow for delegation to overcome the
idea that, the problem that some voters might have
lower competence, and they want to give their vote
to more competent voters.
And then we're just saying this may not necessarily work
unless you're careful about how you do the delegations.
- In having the cap like log in would mean they're should
be a substantially large number of the population who
have good competence, like, cause if I have a very limited
number of people with good competence available,
then we can't impose that gap.
- Right, that's a good point.
So what you're saying is, you know, there could be
situations where we do have a very small, maybe even a star,
where the center has competence one.
In that case I would want to delegate to the center,
but now I'm, I'm, you know, I'm prevented
by this cap, and I would get the wrong decision.
So that's true.
I mean we are taking this very specific viewpoint of
comparing to direct democracy, and from that viewpoint
like that's our benchmark.
That's what we're trying to compete with.
You're absolutely right that in some examples, of course,
you know we would want to remove the cap.
And if we wanted to compare, you know, just allowing
everybody to delegate without a cap to a cap,
we could be doing worse with a cap on some examples.
Right, so that's definitely true.
Cool, so let's move on and talk
about participatory budgeting.
So this is joint work
with Gerdus Benade, Swaprava Nath, and Nisarg Shah.
And as I mentioned briefly, the idea in participatory
budgeting is that typically a city want to allocate part of
its budget according to the wishes of its residents.
So people would express their preferences.
And we would aggregate those preferences to get a decision.
So for example, the kind of things people talk about are
building some community center, building even a stadium,
building park, a bike lane, and so on.
These are the kinds of projects people would vote
on in many of these situations.
So the origin of this idea dates back to the city of
Porto Alegre in Brazil in 1989, so almost 30 years ago.
So they kind of lost their minds and decided to allocate
the entire city budget using participatory budgeting.
And this has been a very successful experiment
by many measures.
So you know there is a lot of work in social science
trying to understand did the
city improve based on that.
And the answer, from what I gather, is yes
according to many different measures.
So this was really the first example, but over the years,
and especially recently this idea has
really been catching on.
So for example, as far as I know, the world champion
is Paris which in 2016 had a hundred million euro
participatory budgeting election and committed 500 million
euro by 2020.
Another example that we will come back to is Madrid
that had a 24 million euro participatory budgeting
election two years ago.
In the US, New York City, for example, had a $40 million
participatory budgeting election last year.
And I just looked up Seattle that seems to have a $3
million participatory budgeting election this year.
So it's not like a huge fraction of the budget.
But it is a sizeable commitment.
And really the feeling is that the commitment
to participatory budgeting is really growing very fast
in more and more cities.
And also cities are expanding their level of commitment.
So again I want to think in a rigorous model
about how to do participatory budgeting.
You know, what's the right way to approach it?
So in this model we're going to have a set of voters
that are voting over a set A of m alternatives.
The total budget is some capital B.
And every alternative a has a cost c sub a.
Now every voter i has a utility ui of a for alternative a
that measures the benefit that voter i derives if we
implement alternative a as part of our set of
implemented alternative one of the projects.
Now for a subset of alternatives x, the cost of x,
the total cost is, of course, the sum over individual
cost of projects in the subset.
I'm also going to make the assumption that the utility
i has for subset x is the sum of individual utilities.
Now I think, this is often a strong assumption,
the tivity of utilities.
But here I think it's well justified, because if you look
at many of these real world cases, the projects
are basically independent of each other.
So there are almost no substitutes or complimentarities
and the tivity kind of makes sense.
Now the goal is to find a subset of alternatives x
that maximizes what's called utilitarian social welfare.
So it's denoted using this notation, social welfare of
the subset x for the utility profile u.
So this is the sum over all voters of the utility i has
for the subset x, the total utility derived by the voters,
subject to the budget constraints.
We want a budget feasible subset.
So this is the goal as defined
by Ashish Goel from Stanford and his team.
They've done very interesting work on both the
theory and the practice of participatory budgeting.
I'll tell you a bit more about some of their ideas later.
But, you know, I guess for many of you as you are
staring at this goal, what you are thinking is this is
just a knapsack problem, where our alternatives are items,
the weight of an item is the cost of the project.
And the value of an item is the total utility
to all of the different voters.
So knapsack is an np hard problem,
but it's pretty easy in practice.
The main obstacle is that we don't actually have
access to these utility functions.
Right we can't really go around asking people, you know,
what's your utility for every alternative,
especially when you think about some of these real
elections where in the final stage we have like, you know,
up to 100 final projects, this would be
very difficult for voters.
So what instead happens in practice is that we ask voters
to express their preferences using some kind of input format
which is a format for their votes, and we aggregate these
votes in order to make a decision
about which subset to fund.
So what I want to do is tell you about a couple of input
formats that people have been talking about.
And I'll do it in the context of this imaginary city,
with a couple of imaginary alternatives.
And I'm also going to take the
viewpoint of one particular voter.
So when I talk about utility, it's going to be
from the viewpoint of one voter.
And I'll show you how this voter maps her utilities
into votes in different formats.
So, one possible alternative is building a town square
which has utility eight to this one voter and costs nine.
We could build a bike lane with utility two and cost one.
We could build a park with utility three and cost two.
Or we could invest in clean energy.
This has utility six and cost six.
Now one natural, you know, way in which to express
preferences is just to sort the alternatives
by order of utility.
So this is called ranking by value which would be
ranking by utility in our case.
And this would put the tunnel square first,
then clean energy, then the park, and then the bike lane.
Now another option suggested by Ashish Goel
and his collaborators is looking at value for money
which is the ratio of utility and cost.
In our case, this would give the opposite ranking
where the bike lane would be first with a ratio of two.
Then the park with a ratio of 1 1/2 and so on.
Now the input format that Ashish and his group are
actually advocating is something they call knapsack voting.
In Europe it's been used independently.
For example, in the Madrid election that we talked
about, they think of it as shopping cart votes.
But the idea is that we just ask every voter
to solve their personal knapsack problem.
So we tell them, you know, here's the budget,
give me the subset of project that's best for you
under the budget constraint.
So for example, here if the budget is nine,
then our voter would take three of the projects,
the energy, the park, and bike lane.
Together they have a total cost of nine
which matches the budget.
And of course, you can do better with a budget of nine.
Now the last input format is something that we advocate
in our paper.
We call it threshold approval.
And the idea is that I'm going to ask every voter
to either approve or not approve a project.
And the voter should approve a project, or check it,
if its utility is above a given threshold.
So for example, if I set the threshold at five,
then my voter would approve the town square and clean energy
and would not approve the other alternatives.
So hopefully these all seem like reasonable, yeah...
- So but the cost is a negative utility so what I would have
done is rank by difference between utility and cost.
- Cost is the amount of money needed to build a project.
- Yeah, so spending money has negative utility.
- So what's your proposal?
- By some, let's say the simplest version is the
difference of the utility.
- Oh difference, yeah, so this is sort of been captured
by ranking by value for money which looks at the ratio.
Right, so here if...
Okay, sure, I mean, so this is a definitely valid idea.
It's not something that, to my knowledge,
anybody has looked at.
I think it's, it's more natural psychologically
to think about ranking by value for money,
because essentially what you would tell voters is what's
your bang for buck.
Right, whereas if you look at the difference,
you know, without having any evidence for this, it feels
to me like something it would be more difficult
to express and think about.
But it's, you know, it's definitely a valid idea that
will again capture the tension
between higher utility and lower cost.
Okay, so, so hopefully these all
seem like reasonable input formats.
And, you know, the question is which one should we use.
So the main conceptual contribution of our work
is giving an objective way
in which we can compare different input formats.
The way we do this is by relying on a framework
that I've been calling implicit
utilitarian voting in recent years.
So one idea at its basis is the idea that every voter i
would report a vote sigma i that is consistent
with her utility function ui.
And consistency is exactly what we saw
in the previous slide, where you can take utility
function and map it.
It induces some kind of vote in a given input format.
So we're going to denote it using this triangle notation.
Right, so we want to look at randomized voting rules
that take as input an input profile of vector votes.
And map it to a budget feasible
subset of alternatives possibly at random.
The main definition here is this definition of distortion
which dates back to work of mine with my Phd advisor
Jeff Rosenschein in 2006.
What it's trying to capture is the idea that we
want to maximize utilitarian social welfare,
but we don't have access to the utilities.
So instead we use the votes as a proxy for the utilities.
And what this is trying to capture is how much we lose
because of this lack of information.
So formally, the distortion of voting rule f
on an input profile sigma is the ratio between the
social welfare of the optimal solution, the one that
optimize social welfare among the budget feasible subsets.
And then in the denominator we have the social welfare
that we're getting when we apply our voting rule
to the input profile sigma.
Now notice that both of these are measured according
to some utility profile.
What's the utility profile?
It's going to be the worst case over all utility profiles
that are consistent with the observables.
Right, for anything that our voting rule does,
we ask how badly can it do compared to the optimal solution
according to the worst utilities consistent
with what we saw.
So as advertised, this definition allows us to
objectively compare different input formats
by associating an input format
with the worst case distortion of the best voting rule.
What do I mean by that?
For every possible input profile, I take the voting rule
that optimizes this distortion.
That's going to be the best voting rule.
And you can define it on every input profile separately
for a given input format.
And then I look at the worst case.
What's the worst case?
It's the worst case over
all possible vectors of observables,
all possible input profiles.
So what this is inherently capturing is how useful
the information encoded in the input format is
for optimizing social welfare in the worst case.
And the theoretical results in the paper focus
on giving bounds on the distortion associated
with different input formats.
So here are the results on a high level.
The candidates are the four input formats we talked about,
ranking by value, ranking by value for money,
knapsack voting, and threshold approval.
And what we show is that for ranking by value,
the distortion is order square root of m,
and similarly for ranking by value for money.
For knapsack voting, the distortion is much worse.
It's order of m.
And this is really horrible, because you can get an upper
bound of m on distortion
by choosing a single alternative uniformly at random.
So this is basically saying that the information encoded
in knapsack vote is completely useless
for this objective of optimizing social welfare.
And in this competition by far the leader is threshold
approval where the distortion is only logarithmic
in log squared of m.
So this says that this information intuitively is
much more useful for optimizing social welfare.
Now a really nice aspect of these theoretical results is
that they're also very good predictors
about what happens in the average case.
So here are some results in the average case.
What we did here is we got real data,
generously given to us by Ashish Goel and his group.
So the data is from elections that they ran in Boston
in 2015 and 16.
Each one of them has a couple of thousand votes.
And we took the real alternatives, the real costs
of the alternatives, and the real votes,
and from the votes we extrapolated random utility
profiles that are consistent with those votes.
So this is where things are random.
And then once we have the utilities, we can translate
them into any input format that we want
and apply different aggregation methods
and see how well the set of alternatives that we get is
compared to the optimal solution on the input for,
on the utility profile that we drew.
So this is what we refer to as average welfare ratio.
You know, just this, the equivalent of distortion
in this average case.
And I want to show you, like a couple of things
on this, in this graph.
It's one thing to note is that the four leaders here
are the four input formats coupled with the optimal
deterministic distortion minimizing aggregation method,
the optimal voting rule.
And what you can see is that they're sorted in the same way
that we had them sorted in the theoretical results,
with threshold approval first, then ranking
by value for money, ranking by value, and knapsack.
Another thing that's interesting is this thing
over here, greedy knapsack.
So this is what was used, for example,
in the Madrid election that I mentioned.
What they do is they do knapsack voting,
which they call shopping cart votes, and then
to aggregate those votes, what they say is every time a
voter puts an alternative in their subset,
they give that alternative one point.
And then we just sort the alternatives by the number
of points and greedily try to stick them into the budget
until we run out of budget.
So what you can see is that even in the average case,
this is doing significantly worse than these
methods designed for the worst case.
So the take home message from this part is that
threshold approval coupled with optimal distortion
based aggregation stands out
as a promising approach for participatory budgeting.
Nevertheless, the jury is very much still out,
even the jury inside my head for various reasons,
one is that if we go back to thinking
about the cognitive burden voters and how difficult it is
for voters to understand what we want from them,
threshold approval is quite difficult to explain.
So we actually have some ongoing human subject experiments
with collaborators in Israel trying to understand, like
how well this works when you actually get people
to express their preferences in this format.
But it's clear that it's not something that's
easy to express to people.
Another issue is that there are other completely different
approaches that we have just started thinking
about, where if some theoretical conjectures are true,
this would give a really compelling
alternative to this approach.
So like I said, you know, this is very much
work in progress.
I feel like really the jury is still out.
But I think this points the way
to some promising directions.
Yeah.
- Suppose we have a certain number of input formats
and how well they do, but ideally what we'd like
to answer is the question what is
the optimum of good format.
Are they all possible?
- Yeah, that's a great question.
So, the question was what's the optimal input format.
And I think to make it a bit more precise, you would need
to ask, or at least like the way I think about it,
is what's the optimal input format if I allow you
some bound on communication for example.
So that's something that we have actually been thinking
about, and we have some results.
But one caveat is that if you, you know, in this case
we have to care about the human aspect of it as well.
So many of these input formats that give you optimal
results or something that would be completely unintelligible
to a person, right.
So there is a, that issue, where even if we optimize this
measure theoretically,
we might do really badly in that respect.
So let me move on to the third part of
the talk, virtual democracy.
So this is based on joint work with my student,
Ritesh Noothigattu, my colleague in the CMU machine learning
department, Pradeep Ravikumar, and a team of collaborators
at the MIT media lab, led by Iyad Rahwan.
So the starting point is this website created
by our MIT collaborators called Moral Machine,
where they tried to understand how people behaved
in the modern incarnation of the classic trolley problem.
So here the idea is that you have an autonomous vehicle
that gets into an inevitable accident.
Let's say it has a catastrophic brake failure.
And some people will inevitably die, and you have to
make the grim decision who to kill and whom to save.
So here's an example dilemma that
a human visitor might face.
You know, a user of this website.
So, you know, one option is driving straight
and killing the pedestrians.
And the pedestrians were crossing on a red light,
and there's one overweight man, one male doctor,
and one baby, or the car can swerve, you know,
take an active action, run into this road block
and kill off the passengers.
And you can get all the relevant information
about the passengers.
The one interesting thing about this particular dilemma
is that if you look at the car, you can see that there
is a baby in the driver's seat of the car,
which kind of makes sense,
because it's an autonomous vehicle.
So, you know, there could be a baby there.
But by the same logic, this must be an autonomous stroller,
because nobody is actually pushing it.
(audience laughs)
And if you thought, like, this dilemma is wacky,
look at this one where we have this daring escape
from the animal shelter with five animals
in the car, and we have to choose between the
animals and the people.
So most people are biased towards their own species
and would choose to save the people over the animals.
So this is a slightly easier dilemma.
But, you know, so this is the type of dilemma they've
shown to people, and like I said their purpose was
to understand how people make this type of decisions.
They've been very, very successful at collecting data.
So at the time of our collaboration, they had more
than 18 million pairwise comparisons
for more than 1.3 million voters.
Right now they have more than 40 million
pairwise comparisons, so even more.
And they, the starting point for our project was
the realization that this vast treasure trove of data
can actually be used to automate this type of decisions
and not just to understand how people make them.
So this has led to this, you know, this framework that we're
proposing for automating this type of decisions.
It has four steps.
So the first step is data collection, basically asking
people for pairwise comparisons,
on the dilemmas that we care about.
The second step is learning.
So here we would learn a model for each voter based
on this, these pairwise comparisons we collected
which will allow us to predict what they would
want on dilemmas that we haven't seen before.
The third step is summarization.
So here the idea is that we are going to have
to make decisions in a split second in running time,
and we want to take all of these individual models
and summarize them as one concise
model of societal preferences.
And finally, aggregation happens at one time.
So we're facing a particular set of alternatives
that are the ones available right now.
And the idea is we'll instantiate our model of societal
preferences for the current alternatives.
And we'll aggregate this model using ideas
from social choice to get, you know, social choice
for virtual voters that will be the choice of our algorithm.
So, you know, what we did in our work was we
instantiated this approach for the autonomous
vehicle domain using a couple of specific design choices.
And this has led to an implemented system that can make
decisions in this autonomous vehicle domain.
So I want to tell you in a few minutes about, you know,
what this instantiation of the approach looks like.
So the first step is data collection.
This was already done by our MIT collaborators,
like I said, collected the whole, the whole bunch of data.
The second step is learning.
So here we use a model for preferences called
the Thurstone-Mosteller Model or the Thurstone Model Five.
There's no novelty here.
This is off the shelf.
The idea is that every voter has some
mean utility for each alternative.
And to compare alternative, or to give a ranking,
the voter will draw a sample from a normal distribution
around each mean and would sort the alternatives
in the order of those samples.
So for example here, I might draw sample for the red
alternative, one for the blue and one for the orange,
and then I would sort those alternatives
in this order and it would give me blue
above orange above red,
even though the orange mean is larger than the blue mean.
Right, now what are the means?
So we take a linear parametrization.
The mean is the dot product of a parameter vector
beta for this voter and the feature vector xi.
So for example moral machine, there are 23 features.
So it's things like are we looking at pedestrians
or passengers, if pedestrians are they crossing
on a red light or on a green light.
Are we taking an active action
by swerving or going straight,
information about the character types, you know,
which characters we have in this dilemma, and so on.
Right, and the parameter vector beta is
what we're trying to learn.
Now, for step three, at the end of step two,
we've learned these 1.3 million Thurstone-Mosteller Models
represented by these vectors of parameters
beta one to beta n.
And now we do something very naive.
Okay, so we come up with one Thurstone-Mosteller Model
for society, where the parameter vector is just the
average of all of these individual parameter vectors.
So we have some theory that says that this is a
reasonable thing to do.
But I think the main justification will be
the empirical results that I'll show you in two slides.
Okay, so step four is actually where almost all of our
technical work goes into in the paper
and most of the innovation.
So here we now have one summary Thurstone-Mosteller Model
for society, and at one time we're given this particular
subset of alternatives x one to x n.
Our model for society, given those alternatives,
would induce what's called an anonymous preference profile
which you can think of as the predicted fraction of voters
that are associated with any particular
ranking over these alternatives.
Now this is a type of an input that you can feed
into most common voting rules.
It's called an anonymous preference profile.
However, there are two major questions here,
one is how do we do this very fast.
How do we apply the voting rule very fast,
because like I said, we may need to make decisions
in a split second?
And the second is which voting rule to use,
because over centuries people have talked about many
different voting rules, which one is the most ethical.
Now we answered these questions
in some generality in the paper.
I just want to show you a special case of the theorem
for the instantiation that I told you about.
So the theorem would say that for any voting rule
that satisfies two basic properties, monotonicity
and neutrality, it would select an alternative that just
maximizes the dot product of the average
beta vector and the feature vector.
So this is great news, because one thing is that it's,
you know, something that's very easy to compute.
It can be computed in a split second.
And second, you know, this says that we don't need
to worry too much about the choice of voting rule,
because most common voting rules satisfy
these two basic properties, and this says all of them
would agree on the outcome.
Yeah.
- The justification for these three things is a little
weak, because most of the time we're not going
to be facing a split second decisions
and also we can do a lot more efficiently than just
going through every one of the millions of votes.
- Right, so in this domain the type of decisions
we care about is when you get in an accident, you know,
who do you kill.
So these will all be split second decisions,
and particular for this domain, you know,
this does seem completely crucial.
You're right that it wouldn't be crucial in every domain.
There is an issue of computational complexity.
So even if it's not a split second, there are very hard
computational problems lurking behind the scenes here,
especially if you have many models.
Essentially, to make a decision, you have to unpack
all of those models and apply and a voting rule to that.
So even if you have, you know, even if you're in some
environment where there is some dynamic interaction,
and I'll give a few examples later, it could still be
to complicated if you don't have a concise model.
So it really depends on the setting.
Like in some others it may be okay.
And in fact, one of the settings I'll talk
about in a minute, does have the property that we're not
worried about summarization, and we actually don't
do that as part of the approach.
Okay, so let me just show you a couple of empirical results.
So this graph pertains specifically to the accuracy of the
summarization step which I promised to tell
you a bit about.
And what we're doing is we're looking at these 1.3 million
models that we've learned.
In one world, we're taking a random subset of alternatives,
instantiating it in all of these different models,
and then applying a common voting rule called verda
and looking at which alternative won.
In the second world, we do the same thing but
on the one summarized Thurstone-Mosteller Model, right,
this naive average thing.
And what we want to see is what's the probability
these two things coincide.
That would be the accuracy of our summarization.
So what we're seeing here is accuracy is on the y-axis,
the number of alternatives is on the x-axis.
And even with 10 alternatives, the base line of
a random guess would be 10%,
but still we're getting 95.1% accuracy.
And you can see the accuracy degrades very slowly.
So this says, you know, at least in this instantiation
naive summarization works well.
That said, I think there's still a lot more work to do here.
There are very interesting technical questions
about how to do summarization more cleverly.
And also it may well be the case that this kind of thing
doesn't carry over to other domains.
Right, so I'm not saying this is the final word
on summarization, but at least for this particular
application, it does pretty well.
So as I mentioned, I think one thing that's exciting
about this approach is that it applies not only
to autonomous vehicles, but to many other domains.
So one example is kidney exchange.
That's a domain that I've actually worked in quite a bit,
but in a different context.
There's a recent paper by researchers from Maryland
and Duke where they take, you know, independently
propose related but distinct approach,
based on social choice and machine learning.
And what they want to do is, you know, we're trying
to find matchings between donors and patients.
And there, they want to break ties automatically
between matchings of equal cardinality based
on some automated prioritization of, of
specific patients, given by preferences reported by people.
So I think that's one exciting example.
Another example that we've been working on,
and this is where I alluded to summarization not being
a big issue, is a food bank domain.
So we've been working with an organization in Pittsburgh
called 412 Food Rescue.
This is a collaboration
with a colleague of mine, Mingh Kung Lee,
whose been working with them for years.
And the idea is that they initially approach men
in order to build an algorithm that would automate
the relocation decisions of incoming
food donations to recipients.
And what we are doing is applying an approach building
on the one that I told you about in order
to automate these decisions in a way that is
ethical, fair, and also efficient to some degree.
So right now we're actually, right now at the stage
of collecting preferences
from stakeholders in this organization,
you know, donors recipients, managers and volunteers.
But really I think the same kind of approach would
be applicable in many settings where you have some AI
and some degree of ethical decision making.
So the take home message from this part of the talk
is that we have built a system that I think serves
as a proof of concept.
That it's possible to automatically decide ethical dilemmas
without formal specification of
ground truth ethical principles.
And I'm emphasizing this last part, because I feel like
that's been the main barrier to doing this kind of thing,
the fact that philosophers have not given us
a specification of ground truth ethical principles
that we can then encode into our algorithms.
So this approach is obviously practical.
And I feel like it gives a way of making credible
decisions, especially when you think about the number of
opinions that are involved,
you know 1.3 million in our case.
Or at the very least, common sense decisions in this
domain that is incredibly tricky for AI.
So to wrap up let me revisit my overview
slide one last time.
So we talked about liquid democracy.
We talked about participatory budgeting.
We talked about virtual democracy.
I've shown you results that I think of as relatively
preliminary in all of these areas, but I do feel
they point the way in promising directions.
And I hope you'll leave the talk with the feeling that
the whole is greater than the sum of its parts,
in the sense that, you know, these are processes
that are happening in the world right now.
And we can really play a role I think, as computer
scientists in thinking
about the evolution of these different pardigms.
And I really feel like the time to do that is now
before various heuristics become ingrained and while
we still have a chance to, sort of take a scientific
viewpoint on shaping these different ideas
and even to some degree helping think
about the future of democracy.
Thanks a lot.
(audience applauds)
- [Anna] So maybe just one or two questions.
- So why not use virtual democracy for everything?
Instead of liquid democracy, instead of participatory
budgeting, if the models are good then we're done?
- Yeah, that's a great question.
So the question is why not use
virtual democracy for everything.
So, you know, this is, I feel like, there are some
inaccuracies that you would inevitably get when you
take this approach of machine learning.
You know, we have various other results that say
how accurate is the learning step and so on.
So obviously this leads to some sacrifice
and some inaccuracy in how you present preferences.
So I feel like the trade off is only worth it when
there are issues like you need to make decisions
in a split second.
You know, you have a car.
Or in the case of 412 Food Rescue, you're just at a stage
where they have scaled to the size, where they just get
so many donations they can't handle them all.
The equivalent of this approach would be to say,
every time a donation comes in, get everybody together,
the donors, the recipients, the managers and so on,
have everybody vote on where we should give the
current donation and aggregate those preferences.
Right, so in situations like that,
that's not something that's realistic.
Whereas this kind of approach does give us some way of, you
know, making these decisions.
- I thought it would be represented by a model of me
than by a representative who has, you know,
that I have an agency problem with.
- Oh I see.
So you're thinking of replacing representative
democracy with this.
I'm not going to argue with you, like I, you know,
I feel like our representatives often do
suffer from significant problems and we can definitely...
In fact I'm not going to argue this point.
- [Anna] Okay one more question.
- A bit more devil's advocate than that.
It seems like a lot of the time democracy isn't
for like social welfare.
It's to prevent like disastrous situations.
So I think about, like, you know, this liquid democracy.
My impression is this, if you go to like the corporate
world and all these stakeholder voting trusts,
or whatever, like, it seems like you could try things
there, but the reason they don't allow transfer votes
is primarily for robustness.
And it seems like, that there's this sort of major
tension once you have, like, rational agents,
because, like, throughout this there's this major
part that we're rational agents here,
and this thought very much,
like it just a maximization problem.
Once we have rational agents, it seems like
everything is a mess.
- Right, right so just to, you know rephrase,
the point is, you know, we often in democracy want
to make decisions in a way that would, for example,
prevent extreme situations.
And one phenomenon we might be worried about, for example,
is what's called tyranny of the majority,
where we have a lot of incompetent people, whereas
we could have had a few competent representatives
that would have made the right decision.
So, you know, this is definitely a flaw
for democracy in general.
We get this flaw also potentially when we choose
representatives or other office holders.
So I feel like it's very hard to overcome, or like
completely escape, these flaws of democracy if you do
any kind of democracy.
And I think what I'm doing here is taking a very
positive viewpoint of democracy, in the sense that
you know also when I'm thinking about virtual democracy
and participatory budgeting you could also say
why do you want the citizens to make those kinds of
decisions, even though we do have some positive evidence.
So I'm taking a very positive viewpoint in that
democracy definitely has its flaws, but it often works well.
And I'm a big believer in the fact that it does
work well overall,
and usually gives us pretty reasonable solutions.
I agree with you that if we did want to implement
direct democracy or liquid democracy, you know something
that is more direct, we would probably need
some additional safeguards in place.
So this approach wouldn't directly address that.
But you could think about doing liquid democracy
according to approaches that will build on some of the stuff
we talked about, and in addition putting
some additional safeguards in place.
For example, giving veto power to particular people.
Actually, i feel like, you know, dealing with this issue
could arguably be orthogonal
to some of the issues we talked about.
- [Anna] Okay well let's thank Ariel
(audience applauds) and take more questions...
For more infomation >> LE GYPSY Quartet At Tango Y Vinos 1/ - May 2018 - Duration: 13:00. 
For more infomation >> Audi TT - Duration: 1:05. 
For more infomation >> VW T6 California - Duration: 0:54. 
For more infomation >> VW T6 California - Duration: 0:54. 
For more infomation >> VW T6 California - Duration: 1:12. 
For more infomation >> 박유천 (Park Yoo-chun)은 보고서 "All lies"|조회수8.212.910 - Duration: 2:44. 
For more infomation >> VW Tiguan Allspace - Duration: 1:11. 
For more infomation >> Seat Ibiza - Duration: 1:06. 
For more infomation >> 반짝 반짝 빛나는 28 일 컴백, '4 인조 체제'|조회수8.212.910 - Duration: 2:31. 
For more infomation >> Skoda Rapid - Duration: 1:11. 
For more infomation >> Kwas moczowy w organizmie – zwalcz go! - Duration: 10:56. 
For more infomation >> Suzuki Vitara - Duration: 1:11. 
For more infomation >> '허리에서 일어나기를 기다려라.'그러나 춤출 수는 없다.|조회수8.212.910 - Duration: 2:24. 
For more infomation >> Rośliny, które zwalczą nadmiar kwasu moczowego - Duration: 6:44. 
For more infomation >> Suzuki Vitara - Duration: 1:11. 
For more infomation >> Kira and Jack Look Back
For more infomation >> Hawaii teen wins big on the wrestling mat - Duration: 3:37. 
For more infomation >> Renault Espace TCE 225 INITIALE PARIS 5P. Pack Winter; Ruimtebesparend reservewiel (max. 80 km/h) - Duration: 1:13. 
For more infomation >> Volvo 940 2.3 STATION AIRCO TREKHAAK POLAR - Duration: 1:12. 
For more infomation >> Peugeot 308 SW 1.6 THP XT TREKHAAK CLIMA PANORAMA DAK - Duration: 1:11. 
For more infomation >> Fiat Grande Punto 1.4 Edizione Blue & Me 5 Drs AIRCO CLIMA - Duration: 1:08. 
For more infomation >> Für Meghans Hochzeit: "Suits"-Stars auf dem Weg nach Windsor ! - Duration: 3:03. 
For more infomation >> Mackay | Mi angel de amor | Ft Key-K | Prod By @PharaohCorps - Duration: 2:45. 
For more infomation >> Sunshine Stenciling (New release from Simon Says Stamp) - Duration: 6:12. 
For more infomation >> Killer Karaoke arriva su Rai 2, Gigi e Ross conducono la versione italiana - Duration: 4:03. 
For more infomation >> Striscia la Notizia difende Barbara D'Urso, una trappola nei suoi confronti? - Duration: 4:03. 
No comments:
Post a Comment