Thoughts on Technology

Changes in User Behavior: How 3D Graphs can Provide Deep User Insight

Posted by | Thoughts on Technology | No Comments

During my time at Localytics there was a drastic movement towards getting deeper insight into the customer lifecycle (acquisition to engagement). It’s the holy grail for any marketer to be able to understand where the user was at in their journey, where they might go next, and when the may potentially fall off. This made me construct a theory around how we look at user data so that we can understand better what forces push a user in one direction or pull a user in another. This theory breaks down into 2 major components: Track user behavior in 3 dimensions instead of two, and utilizing deep learning networks to provide insight into where a users’ behavior is moving towards.


In many (if not all) of the current platforms on the market that offer some level of personalization or optimization, they are viewing a single marketing channel in a two dimensional space. What I mean by this is that, for example, we would look at something like conversion rate vs. time on a website. We could even get a little bit more technical and say conversion rate by a unique user over time to get more granular. There are a very select few amount of platforms providing a holistic (barf buzzword) user centric view of how individual users are interacting with each of the individual channels.

As enterprises shift dramatically towards what everyone is calling “digital transformation”, there’s a specific trend that is being surfaced: brands need to be where ever their users are at in all distribution mediums. What this means is that you’re not mobile first, web first, or social first, but rather taking the stance that your users will engage with your brand on one of many different channels, and on those channels in many different forms. For example, a user engaging with the “social” channel could mean Twitter, Pinterest, Facebook, or some random forum with their action being a like, a comment, or a share.

If we go back to viewing marketing channels as two dimensional spaces, we can start get some idea of what this looks like and the X or Y axis. For example:

  • X-Axis = Time
  • Y-Axis = Conversion Rate
  • Graph = Individual User Level


Today, you would do this for many different channels then try to discern what is actually happening. I’ve seen this manifest on Excel spreadsheets where the rows look something like:

  • Table = Campaign
  • Time Series = Week over Week on 6 month basis
  • Mobile App = 3.5% conversion rate on “X” event/trigger
  • Website = 3.2 minute avg. session length
  • Twitter = 2 #’s or @’s referencing “X” brand
  • Email = .03% open rate
  • etc.
Campaign ID Mobile App CR% Website Twitter Email Push
This Week 1995803 3.5% 3.2 min 2 Interactions .03% OR 10% CR
Last Week 1994068 3.2% 3 min 1 Interactions 1.5% OR 9% OR

Listings of metrics like this raise questions such as was this campaign successful? Did our messaging turn off our users or increase retention? Did any users churn week/week? But what if some users underneath a campaign show signs of churning but, in reality, they’re really just not engaging in your emails? Were there outside factors influencing these campaigns, such as weather or holidays? Were the results that I’m seeing statistically significant enough that I can trust them?

There’s a ton of open ended questions that marketers get stuck with and have no clue what any of it means. It’s complete data overload and what they end up doing is surfacing the deltas between the weeks as their KPIs, cherry picking the best, etc. It’s not a good situation overall.


The theory goes like this: When doing aggregate metrics or trying to understand multi-channel marketing efforts, viewing users on 3 dimensions instead of 2 surfaces much deeper insights. This manifests itself into 4 quadrants that generalize the overall user behavior that can occur on any channel:

  • Highly Engaged (top right)
  • Engaged but Not Responding (bottom right)
  • Responding but Not Engaged (top left)
  • Not Engaged and Not Responding (bottom left)

We want to keep them as generalized quadrants because there needs to be ample room for interpretation based on channel, vertical, or business interpretation. In my experiments, it’s easiest to contain the graph on a scale of 0-100. We’ll come back to this later.

Here’s the hypothesis that we’ll work off of, viewing the above theory from an app perspective :

If we think of audiences/users in 3 dimensions with more aggregate-type metrics, we can provide a more prescriptive insight into audiences/users based on many summary metrics. Additionally, we can surface audience/user movements automatically without the need for customer input.

We have our hypothesis, but now we need to add the general titles for the X and Y axis. This looks something like:


On our X-Axis we have App Engagement which could be an aggregated metric based on many different individual values. On the Y-Axis we have Marketing Engagement which could be attributed to all the different mediums a user might interact with when on their mobile device. We also have our 4 quadrants which help us positions users. For the above graph, I swapped out “Not Engaged and Not Responding” with “Risk of Churn” which can be viewed as synonymous.

In this world, new users would be placed onto the chart in the very center and, based on their actions, start to form a vector for their behavior. So once you start to have users on the chart, it may look something like:


With many analytics or marketing platforms, there are APIs that allow you to pull data on a scheduled basis. In our scenario here, we would want to pull data from many different data sources within our ecosystem into a nightly or weekly snapshot. This could be a super flattened JSON file that stores performance data for campaigns, segments, audiences or individual users. We don’t want to pull very specific data such as when a user engaged with a campaign, but rather aggregate or composite metrics (ie. 7 day retention).

This nightly or weekly snapshot of composite metrics allows us to identify anomalies, user behavior shifts over time, and assign a “vector” of movement. From there, the vectors of movement can be charted to look something like this:


Now we’re getting somewhere interesting. On each of the axises you can see different types of metrics that may contribute to the aggregate or composite metrics. Since we now have day over day or week over week deltas of user behavior, we can plot vectors. Different vector deltas may have different positive, neutral, or negative indicators associated with it. In the above example, we have 2 green users who are in the “Risk of Churn” section but they may have both elicited a significant change in user behavior based on the aggregated metrics. These changes may be signifying that the user is being recovered and coming back towards a more healthy state with regards to App Engagement and Marketing Engagement.

By doing this, it can help us identify anomalies, thresholds where we may want to intervene with an outbound marketing reach, predict where an audience/user may be moving based on their vector, and get a better overall view into where users may be at in their lifecycle journey. Up until now however, we’ve viewed this on 2 dimensions. If we add a 3rd dimension to our chart, it becomes much more interesting. Let’s assign an App Goal as the Z-Axis which could be something like “Increase App Engagement”, where the metric looks at any increase in engagement as a positive influence. This starts to look like the following:


Now that we’re viewing this in 3d, it gives us the opportunity to see where a specific user may be on each of the axises that we care about. For example, we may notice that the blue user in the top right has about a 7.5 on Marketing Engagement, 7 on App Engagement, and 7 on App Goal completion. This user would be considered a safe user and is stable in what we’d considered the “Highly Engaged” quadrant. However, on the bottom with our turquoise user, we’re seeing trouble. The user is low on App Goals, very low on Marketing Engagement, but has decent App Engagement. If we had a vector assigned to this, we could see the direction in which the user is heading to determine if this is a user who is moving into the “Potential to Churn” quadrant.

How A Deep Learning Neural Network Fits

Here’s where we will go down the rabbit hole. In the above image, our users were defined as “spheres” visually. What if we actually viewed them mathematically as spheres? Follow me on this one.

A user has a sphere around them that defines the behavior quadrants which, in any direction, have a global maximum value of 1. The user initially starts out with a neutral value of 0 (center of the sphere). When the user performs an action such as having 3 sessions on an app in 1 week, we see that as a positive behavior and assign that as “Highly Engaged”. On our graph above, the user started at the center of the graph (5, 5, 5) with the user behavior is at plotted at 0, 0, 0. Now, with the new positive session count towards “Highly Engaged”, we weight this direction in order to provide its vector movement.

Metaphorically, this is similar to having a sheet that has our above quadrants on it that is pulled tight. You drop a weighted ball in the “Highly Engaged” quadrant and the sheet is pulled in that direction. If you wrap that sheet into a sphere, you are doing the same thing except that each drop (or throw) of the ball is pushing the entire sphere in the direction of the quadrant (in a 3 dimensional space).


Credit to Michael Nielsen and his blog on Neural Networks and Deep Learning

We do this through the use of a deep learning neural network. This is supremely described in detail by Michael Nielsen on his blog which I highly recommend reading. In essence, the neural network utilizes many different data sources in order to product a mechanism called gradient decent within a specific “perceptron” or “neuron”. These neurons work on a sigmoid function with their output being a value between 0 and 1. As the output values come through to our “user behavior sphere”, the values weight the sphere in the direction of quadrants that the user behavior is attributing towards. This moves the “user behavior sphere” into that quadrant.

For example, positive App Engagement and Marketing Engagement pushes the user sphere towards the “Highly Engaged” quadrant while day over day or week over week reduction in interaction on either channel conversely weights the user sphere down towards the likely to churn.

With all of the different channels a user may be interacting with a brand on, a neural network can ingest this data, decipher it, and provide the proper vector for the user with respect to our 3d visualization.

Bring it back together

If we climb out of the rabbit hole and out of an app specific example, we can start to view that a 3 dimensional view of users can be beneficial for synthesizing and expressing user behavior across many channels as one unified view. It allows us to do interesting things with the concepts of explicit vs. implicit user behavior and be able to plot that behavior in a way that makes sense to marketers as well as machines. To be clear, I don’t think it’s beneficial to surface the above graph visualizations to marketers but rather view the data from this perspective. What I do think is that, behind the scenes, we view the user behavior in this format mathematically but surface up at a dashboard level only the interesting movements, such as “X number of users are potentially moving from one quadrant to another”.

This theory could be useful for complex ecosystems that have disparate metrics and data sources. However, it is important to note that I believe that this could only be useful when the metrics are in an aggregated format. I believe that if you did this on an individual metric basis that it will dilute value since we’re not taking in the full picture of the user or audience on all channels.

I still have many questions personally around how this can be applied to different business models given that metrics like engagement mean completely different things to different verticals (media vs. travel). Additionally, this may only be useful at an audience level instead of user level due to the graphs nature to simplify and abstract away the metrics.

We currently do similar types of thinking around users when it comes to radar graphs. We assign a metric and value ranges to each of the points on a radar graph and track user behavior as they progress towards “who they are”. This is especially useful for when we’re trying to understand what persona a user may fit into. However, the drawback is that this is still 2-dimensional thinking and has flaws such as not being able to effectively measure goal completion or aggregate metrics effectively (7 engagement, weekly conversion rate, etc.). There’s a world where we could add a “depth” factor to a radar graph but that is a theory for another day.


Radar Chart example from Chris Zhou

Additional/Upcoming Ideas

I have more thoughts around how this can be morphed into interesting things like open graphs for further automation among the entire marketing ecosystem. For example, if we have a user who is starting to become low on certain marketing engagement efforts, could the system automatically create a retargeted marketing campaign on Facebook as a method of exploitation test to further optimize the system?

A very open area that I need to consider more here is specifically around downward/negative weighting. In reality, users don’t necessary do events that show that they are likely to churn, except for some exception like opting out of push notifications. Users often elect to “churn” simply by stopping to use a channel. This is problematic for this model as we don’t have a proper way of giving the graph empirical data to weight it down. My hypothesis here is that we have a time decay model that looks at the deltas of values day over day or week over week to identify shifts in engagement specifically. This decay model would take into account metrics like % difference of engagement, session length decrease, # of days/weeks of downward interaction. The output would be a weighted value pulling the user sphere towards a non-favorable quadrant.

Another area of thought exploration is around doing further clustering within the quadrants so that marketers can target certain sections of the quadrants. For example, a marketer may see that one of their users in the “Highly Engaged” category is starting to move towards “Engaged but Not Responding Quadrant”. If we view the graph on a scale of 0-100, the marketer may want to intervene when the user has moved into the region of:

  • X-Axis = 60-65
  • Y-Axis = 55-65
  • Z-Axis = 50-60

In an ideal world, either the system or the user would set these quadrant sub-parameters and, when they log into their dashboard each day, get a notification with how many users have a “drift vector” showing that they may move quadrants. This would allow the marketer to get ahead quicker on the changes.

Yet another area of contention is the quadrant naming. I think there’s a good argument that would need to happen on whether or not the quadrant naming, nomenclature, and conventions make sense.

It’s a crazy theory but, based on the work that I’ve done in my past with different teams, there is some validity with regards to how marketers view users and how we view data. More thoughts to come

Love the idea? Think I’m crazy and full of shit? Leave a comment and tell me what you think!

7 Things I’ve Learned Recently

Posted by | Thoughts on Life, Thoughts on Startups, Thoughts on Technology | No Comments

Life is pretty interesting with the direction that it will take you, who it will introduce you to, and the lessons it will teach you in weird ways. I recently had some pretty impactful changes and figured it would be good to share my recent learnings.

You can never be to cognizant of your business health

In the past few months it has become apparent that when you run a company, no matter the size or growth you’re experiencing, you can never be too close to the core fundamental business metrics. It’s important to keep an eye on critical areas of revenue, profits, growth, retention, and reduction of complexity. Having a core vision that you have conviction around is paramount for bringing people together as well as providing a lighthouse, dictating what metrics matter the most.

Be open to meeting with unique people

I’ve recently started saying “fuck it, let’s meet” more to random strangers who reach out. I’ve done the same as well by randomly reaching out to people in different industries to learn about their work, their successes, and their failures. I’m not sure what has recently triggered this but it has been incredibly enlightening.

For example, I went to a bioinformatics MeetUp recently and learned a lot about what challenges bioinformaticians face with their day to day work and what matters most to them. Turns out, having open data within the science community is really important. Who knew! Apart from that, I’ve met some individuals recently who have challenged my frame of thought and helped me develop a different framework for thinking about problems.

My take away lesson was to put yourself out there and talk to people you wouldn’t have otherwise – especially in different industries. I’ve found that understanding how they solved their own industry problems can translate into solving your own industry problems.

Be confident about what you’re worth

Pretty simple learning here: don’t devalue what you believe you can offer to others. They may not see the same type of value which can either tell you a lot about the person or provide a conduit within a conversation to explain how valuable you are. This isn’t to say that you should be cocky, but I’m finding that it’s important to have confidence in yourself, believe that you’re worth what you think you’re worth, be persistent and consistent in that confidence, and implore that on others who may not see the same way.

Life is pretty short

I know everyone says this but seriously, god damn does it feel short. I know this is a typical thing for someone to say and dumb to reiterate at my young age but… it really feels short. I think about where I was two years ago and it is shocking how fast things can change. However, I was reminded that life is also long, depending on your perspective, goals, and ambitions at that current time. What’s short now may seem long in the future. But for me, right now, I can’t seem to get enough time on my side.

Graphs are probably the future of computer to computer interaction

I had a great conversation recently with a colleague who is exploring different types of graphs as a method of computer to computer communication. In essence, individual components have their own operating rules and jurisdictions that they behave by. However, when interacting with other components, they have contracts that can be extended in order to have a meaningful “conversation”. This is critically important when working in increasingly complex situations like supply chain management, ERP, and other enterprise business functions.

Furthermore, with the internet of things coming into force, this complex graph of “things” will need to have a concrete way of sharing data with each other. You don’t want to have a situation where a street light isn’t talking to a car giving it instructions only to find out that the car isn’t talking the same language. That will end up bad.

The core principle is that components can have many data ingestion points to make decisions but have binary outputs that interact with other components that make binary decisions based on lots of data points, and so on. The method for this is called Intelligent Agents and is a subset of development methodologies underneath machine learning. I’ll probably write an expansion on this topic soon.

Every bad or tough situation has massive opportunities

Being put randomly into these situations sucks because it’s pushing you into the unknown. I hate the unknown. I’m always trying to be 100 steps ahead and super calculated. But alas, shit happens and what matters is being able to react in a flexible manner. Instead of victimizing yourself, I’ve found it better to look at the situation as “Now that I’m here, what are the good things about this situation? What unique opportunities do I have that I didn’t have before?”. Always look for the positive, even when shit really, really sucks. I’ve found that things typically course correct themselves and opportunities start opening up usually in less than a week. This all comes back to networking and being confident. Trust yourself in finding a way out of the shitty situation, even if the way out is completely unclear.

Saying no to features is hard. Finding features that are game changing is harder.

Both are difficult tasks of a product manager. You have lots of competing priorities and objectives and your job is to act as a shit umbrella and a filter. You have to have extremely firm but loosely held opinions about the world. It’s hard to say no to features that don’t align with your long term vision. If you cave in to those feature requests, they’re going to come back and bite you in the ass.

Even more hard is finding the features that are really game changing for customers. This is what makes a good product manager valuable. Often times the answer isn’t clear and the customer doesn’t know exactly what they want. It’s a 50/50 balance of solving their pain points now but also showing them the future. I advocate solving customer problems but really think that doing this constantly doesn’t progress you as fast as it could. If I constantly solve for a 10% gain, then I’m competing with the market (you need to do this because it’s the bread and butter). However, if I’m able to give a 10x solution to a problem, I’ve leapfrogged the market significantly. Often times, solving for 10x is incredibly unclear.

These are probably pretty obvious, hippie-millennial talking point and learnings but they’ve recently been really reinforced in the past couple months for me. Things are often not as bad as they seem initially.

Federated Queries in Genomics

Posted by | Thoughts on Genomics, Thoughts on Technology | No Comments

In the late 90s and early 2000s there was a trend called “federated search” that made some ground in how the web was architected. Federated Search, more commonly known as “federated querying”, allows a system to query multiple disparate data sources in order to surface up a single result from a single query. It never really made grounds however due to the disorganized nature of where we were at with data architecture as well as data storage performance.

One of the other trends within this world was storing a bunch of data into a single database from many different sources which is where the term “data lake” came into play. The term describes exactly what it is: a giant lake of data that you can query against from many different sources.

These were two trends that ended up dying pretty quickly as optimal solutions became apparent for specific industries. Postgres and NoSQL moved on to the scene and became the choice database solution for many needs within the high tech world. With NoSQL specifically, it’s performance was incredible for fast ingestion and really fast query times. However, the unstructured nature of NoSQL causes it to be problematic in many types of organizations.

In a funny way, the “big data” world is coming around to going back to federated querying. I think specifically that Genomics is going to be a huge user of this type of architecture given the nature of dramatically different database requirements for different parts of the genomics pipelines. Some systems will want very rigid and structured databases whereas others will want the freedom of unstructured storage that allows them to scale and bend data.

For example, you might want to store information about Genes and all their annotated meta data in a MySQL database with optimized query performances. However, you may want a Postgres build for ingesting human genome variant data. Postgres could be nice for this given its parallel processing nature. In another realm, you may want to store clinical trial data inside of a NoSQL database in order to return large arrays of data extremely fast. This means that you have 3 different databases with, while similar, different query languages. A structured federated query language to hit all 3 sources would be beneficial.

An example of a federated query could look something like:

PREFIX gene: </local_endpoint/genes>
PREFIX diseases: </local_endpoint/diseases>
PREFIX genome_variants: </local_endpoint/genome_variants>
PREFIX clinical_trials: </local_endpoint/clinical_trials>

SELECT ?genes ?chromosome ?basepair WHERE {
    SERVICE </local_query_api_endpoint/> {
        ?basepair BETWEEN 100000 AND 200000:genome_variants.
        ?chrom = 11:gene.
        ?diseases genes:sameAs ?genes
        FILTER(str(?ADHD), "DRD4"))

This isn’t necessarily the most pretty (or accurate) representation of a federated query but it provides the structure in which we might join together a multiple data sources with specific queries into one table. This table might be persisted or we may just be doing a floating query, in which we could store the query in a temp table.

Apart from its flexibility, federated queries are attractive to genomics primarily because it can provide incredible performance across massive datasets. There are other query languages or implementations in the field that act as more aggregators vs. federations however their purpose is very similar. Facebook, in my opinion, has the most advanced version of this where they use a node:leaf system pair with GPU based operations for querying against petabytes of data.

Among the other little experiments I’ve been running, I’ll be testing out an implementation of an aggregation service on a small scale from disparate data sources to see if I can create a simple example of this.