Over the past 4 years, I have gone through periods of installing and uninstalling Facebook’s mobile app out of frustration and protest because of reasons you are all likely familiar with:
- why is my battery dead!?
- why is my timeline so messed up!?
- why is my private post viewable by everyone on the internet!?
- why do these these security settings keep changing!?
- why is the mobile app nearly 1/2 a GB!?
will find these complaints reasonable, but, probably not compelling enough to stop using Facebook’s mobile app. They have their reasons, I have mine.
My most compelling reason to use Facebook is communication with my immediate and extended family because it is their preferred method vs. the telephone.
But this past winter I found my final reason to stop using Facebook’s mobile app: it was listening to my private conversations!
Despite numerous claims to the contrary, this post is about my experience with Facebook targeting ads I believe were based on conversations held in proximity to my mobile device.
Last year my wife quit her job and went back to school to study a new field, one that I was not familiar with or personal interest in.
In January, she and I sat in our living room talking about her day at school with my phone on the table, in the open, between us. She excitedly told me about a new company she just heard about that she wanted to purchase products from to help her start her business. The company name she mentioned immediately struck me as unique, so much so that I repeated it in a silly voice because it was phonetically fun to say.
About a day later, I logged into FB using my desktop browser and the very first advertisement presented to me was an exact match for the company my wife mentioned in our conversation. I recognized it instantly because the advertising was for a category of products that I have absolutely no personal use for, and, the name stuck out to me as being unique.
The Test Environment
This information is also pertinent to my findings:
The hardware (in proximity to conversation):
- My iPhone (Facebook installed)
- My wife’s iPhone (no social media apps or accounts)
- Samsung TV (audio features disabled)
The system software and applications:
- Facebook mobile app – installed on user’s device, placed on a table between user and another during conversation.
- Facebook platform – highly available processing and storage tier
- Facebook web app – accessed via laptop, in a different room on a different floor the house.
The user himself:
- one-time commercial broadcast radio voice-over artist for radio advertisements and promotions.
- one-time analytics dashboard developer for presenting insights into content targeting effectiveness.
- long-time meta-forward-observer and mostly hater of ALL advertising (except the 0.001% that are actually well written or funny).
- curious, data-driven do-gooder, snark
The audible data:
- company name was not a ‘dictionary’ term found in regular or common speech.
- company name was a mashup of two words (likely for distinction and because the web domain was available for their digital marketing).
- company name was easily distinguishable from other common phrases my wife had been using during her course of study.
- company name was never once previously mentioned in our private conversations
- does not have radio or television advertisements broadcast in my area (or on any streaming services we use)
- markets to clinicians
- I have a Facebook account
- wife has never had a FB account
- wife is not in my interest graph
- company sells products I have zero interest in and never sought
How I noticed I was targeted was simple:
- my wife spoke aloud a unique word (signal input)
- in a conversation with my phone present (sensor)
- which had Facebook software installed (platform client)
- the word was never previously spoken or searched for (high cardinality, not in my graph)
- platform matched advertiser keyword with my input (signal processing)
- platform served me an ad via desktop web browser (web application client).
My “accidental” black box test
of Facebook’s advertising system indicates the mobile app software is listening to my conversations. This methodology is called Observability
(a measure for how well internal states of a system can be inferred by knowledge of its external outputs) which is a topic in Control Theory
, which is the interdisciplinary study of dynamic system behaviors. Less formally, this means that from the system’s outputs it is possible to determine the behavior of the entire system.
To investors, I believe Facebook’s interest graph content targeting system appears to be working as designed, and, quite well. You should be pleased.
My job for the past 20+ years has been to observe and verify applications and their systems work as designed.
In my particular engineering role the single most critical skill to develop and hone is observing every action (and reaction) of a system while operating a software application. After two decades, it becomes a near constant state of mind which is how I connected these dots.
Caveat: I could be wrong. I have not done the forensic analysis of the code or the devices. I am writing this based on my many years of experience finding and isolating system behaviors like this one I observed. I’m also relying on numerous sources of published statements and research material to support parts of my claim.
Device Information section describes how they fingerprint my device, including it’s location (especially in relation to “specific” locations) using GPS, Bluetooth and WiFi, depending upon permissions I’ve granted.
“We collect information from or about the computers, phones, or other devices where you install or access our Services, depending on the permissions you’ve granted. We may associate the information we collect from your different devices, which helps us provide consistent Services across your devices.”
Even though the professor’s claims (links above) have been debunked, Facebook admits it did but now “only listens for 15 seconds while you are posting.”
So why continue writing this article?
- I was not posting during my conversation with my wife (phone was not in my hand)
- I intentionally removed FB app access to my device’s microphone (long before this incident)
- I am also pretty darn sure I was logged out of the app.
Additionally, it’s possible to “listen” to audio within proximity using other device sensors besides the microphone.
Before you go thinking I have a collection a tinfoil hats for every occasion, there is a plethora of material produced by much more qualified researchers and engineers than I on the subject of tracking users via mobile phone sensors for content targeting. They have published, presented and even demonstrated on the subject. They are also questioning why device makers allow apps direct access to sensors.
To start simple, eMarketer estimates that Facebook’s platform will generate $36 billion in net digital ad revenue in 2017, up 35% from last year, giving it the second largest share of the global online ad market with 16.2%, behind Google’s 33% (double FB’s take). High volume ad targeting platforms like Facebook and Google are not unlike the scheme in Superman 3 when Richard Pryor’s lovable character hack’s his employer’s mainframe to collect the salami-slices of a rounding down routine
on everyone’s paycheck. More on the market opportunity later, let’s stay focused on the tech.
One of the most interesting and compelling technical sources I found when I started researching direct access to hardware sensors via applications came from a Stanford University which describes how mobile applications are given direct access to hardware sensors. In particular, their research proved your phone’s accelerometer can ‘listen’ to your speech.
– Project with Videos (very nerdy!): https://crypto.stanford.edu/gyrophone/
– White Paper: https://crypto.stanford.edu/gyrophone/sensor_id.pdf
– Github: https://bitbucket.org/ymcrcat/gyrophone/
– Example Android Application: https://crypto.stanford.edu/gyrophone/application/gyromic.apk
What is important here is that it’s not your microphone (the obvious choice that requires your approval) being employed to “listen”, it is that ‘other’ non-obvious sensor with the funny name that is carrying the water here.
To give you a metaphor, your device OS governs whether or not applications can acquire access to sensors, however, if the application convinces the user for permission to use a sensor, the user can override the OS setting. That’s like the irony of the FDA requiring labels showing active or regulated ingredients – there is no law to requiring non-regulated (aka unknown and unlegislated) ingredients to be listed. More importantly, the FDA is not always up to date on the latest technology, manufacturing processes or impact on the consumer a food or drug may have.
Where things get a little scarier is how some applications are able to obtain direct access to a sensor without any further governance.
To clarify my point, this isn’t Jason Bourne/Snowden stuff. This is all squarely in the realm of technology used today by web services and social media for content targeting.
As I mentioned above, digital advertising is the primary revenue engine on the internet – you (unwittingly) give up your privacy to post cat pics and they return the favor by trying to sell you Jucero and artisanal mouthwash…and you get none of the proceeds for having also providing your attention span. Of course there is a subscription revenue engine too but content “firewalls” have proven to be daunting to implement in a way that is both usable and converts users into subscribers.
Regarding the advertising “spend” (budget), one forecast for 2017 indicates the digital (internet) ad spend will surpass the television ad spend for the first time in history. Yes, we’ll still have glitzy/campy Super Bowl TV ads to watch next February, but, the largest slice of the ad spend pie will go to digital – this is a big deal
Why? Let me unpack this a little: the overall ad spend by medium is typically broken down as follows:
- Digital (internet)
Now let’s look the American advertising industry medium timeline:
From an ad spend perspective, digital is experiencing warp speed growth over the other mediums, but that’s not the half of it. Tech has convinced advertisers to trust them with delivering their ads to the right, best and most important eyeballs.
There is additional benefit to keeping the content cheap but that might affect each business differently.
The scale of ad spending is massive.
This year’s total digital ad spend is expected to exceed $70 billion in 2017. A sub category of Digital is RTB (Real-time Bidding) or “programmatic” and it has skyrocketed over the past 2 years alone. This is the Superman 3 business model again, especially as it relates to automated high-frequency trading models.
According to eMarketing, “nearly four of every five US digital display dollars will transact programmatically in 2017, totaling $32.56 billion. By the end of the forecast period, that share will rise to 84.0%, leaving little doubt that buyers and sellers are continuing to invest in automated ad buying.” All this is despite a major controversy with Alphabet’s YouTube policies that saw major declines for content publishers.
companies invest millions in technology that can more accurately target you because it directly affects their bottom line.
And who is dominating in their ability to not only receive those dollars but make sure advertising feel they are well spent? Google and Facebook. They are spending billions in research and development of not only the technologies they currently employ but also new technologies that don’t even have a market developed for them yet (like VR) so they can be there when our eyeballs and attention spans show up.
Recently there was a lawsuit that tried (and failed) to do something about this practice of tracking users after they log out (unrelated to snooping on conversations).
“The decision, filed late Friday in California, gave Facebook a win in a lawsuit that accused the company of improperly tracking users’ Internet usage between April 22, 2010, and September 26, 2011, even after they had logged out of their Facebook accounts.”
Additionally, the decision said the plaintiffs failed to establish a “realistic economic harm or loss” stemming from Facebook’s comment.
Sadly action failed but it also goes to show you how well cloistered a company like Facebook must be to keep their distant 2nd place spot on the ad revenue trough.
I quit Facebook Mobile apps the day I received the advertisement for the company my wife was so excited to tell me about.
From this point forward, I will only login very ~infrequently~ from a desktop web browser and not on a mobile phone, if I feel like it. It will mostly be to help manage other folk’s projects and to let you know that I am “alive” (btw there is an amazing, real world outside of FB, I’ll post some pictures for you as proof)
If you really need to get a hold of me, you probably already have what you need to reach me (we out there).