Technology

Anomaly Detection – A Novel Approach

Posted by | Enterprise Software, Optimization, Technology | No Comments

One of the harder things to do in monitoring system health or even brand health is to detect anomalies or “events” that are happening that may be out of the ordinary. It gets harder to detect such events when you’re data fluctuates frequently or when you’re trying to build a model that can be applied towards dramatically different datasets. This topic has been talked about many different times and contested with different theories, mathematics, and approaches such that we don’t create “alert fatigue”. Of course, I had to try and do it differently! Let’s talk about the approach I’ve been testing out.

TL;DR – I’m testing out a model that looks at the velocity vector moving average and derivative moving average. By looking at 3 time series data points of the derivatives in the past and extrapolate into the future, paired with the velocity vector, we get a good idea on when an anomaly may be happening.

I’ve explored many different ways including sophisticated machine learning methods. However, one afternoon I had a thought about looking at the problem a different way. This approach includes methods from day trading, physics, and calculus. The approach is simple enough: look at the change in slope agains the moving average. The reality is that there is a lot more to get it to work. And now, for the deconstruction…

Acceleration Moving Average

The first portion of this theory is to take a look at the acceleration moving average. This is found often in day trading as an indicator to a dramatic shift in direction that outpaces the prior accelerations. In this portion of the formula, we look at the acceleration formula as follows:

a = ∆v/∆t

For each time series, we store what the a is calculated out at. From there, we compare that against the moving average. Internally, we have tested out looking at a 14 day moving average on a 10 minute time series. So for each 10 minute increment, we look at the current and compare it against the moving average. However, as you can imagine, this can fluctuate quite dramatically and cause alerts to be sent that shouldn’t be. The risk of looking specifically at this is that you set a static threshold – ie. if current acceleration is greater than acceleration moving average by 20%, send alert. Where this really breaks down is when you get multiple spikes over the course of a day with each subsequent spike being less in volume (but still notable). Since the moving average will increase to account for the most recent spike, you lose out on the sub sequent spikes. Example below.

anomalies

If you look at the large red line right at around 12/1/15, you’ll notice that if we were to use a moving average that our moving average line would be pulled up dramatically. This causes the subsequent events happening at around 12/15/15 and 12/18/15 to be missed. While the acceleration moving average is a novel approach, we’ve actually found that it isn’t necessarily as useful as we’d like. It has often been led astray with wild fluctuations of volume and has a high propensity to trigger alerts that are not actually needed – such as the above. This led to look at a different approach.

Velocity Vector

Vectors allow us to quantify an object’s direction and magnitude. When looking at an anomaly, we want to understand it’s direction of movement on an x,y axis then pair that with the magnitude of volume. We could arguably get rid of the acceleration moving average at this point as they effectively become the same thing once we look at the moving average. Now, the velocity vector gives us a bit of understanding in real time what is happening to our volume. See example below.

velvec2

When analyzing twitter volume, volume can be sporadic. Even when reviewing the velocity vector moving average against the current, we still find that alarms are triggered more frequently than we’d like. This is primarily due to the data not being smoothed out. Meaning, we get snapshots of volume at different time frames as whole numbers, such as 10, 50, 34, etc. This makes it hard to discern the significance of a change in the vector portion of velocity vector. This brings us to the third portion of the formula.

Fourier Smoothing

Since Twitter volume data comes in as chunks of whole numbers, this causes our vectors to change dramatically which renders the prior useless. Vector velocities appear to really only be useful when the data is smoothed out between the actual time series counts. For example, if we have the two data points of 1 and 5, we’d actually want to fill in the difference with 1.1, 1.2, 1.3, 1.4, etc. In an interesting way, Twitter volume data can sometimes look like audio signal data, the sense that it can be incredibly choppy. In order to smooth it out, we can actually use Fourier Smoothing to create a nice looking data set as the Twitter volume count comes in. Below is an example of Fourier Smooth, where we look at discrete values of temperature by day and smooth out the data using this technique.

fourier-smoothing

Now when we look the velocity vector moving average, the value becomes more stoic and doesn’t change nearly as much as it did when no smoothing was applied. If we look at the velocity vector on 10 minute increments as a 14 day moving average, we get some nice insight as to the different fluctuations happening. However, we’re still looking at the current state and still don’t have a good way of letting the machine tell us not only when to trigger something, but letting us know when something might happen. In order to solve the predictive portion of that problem, we looked to derivatives.

Derivatives

Since our Fourier Smoothing of the dataset provided nice hyperbolas, we can easily calculate the derivative of any data point at any given time. In our environment, we have tested out looking at the derivative at each 10 minute increment. Since calculating the derivative gives us a line that theoretically extends both into the past and future, we actually look up to 3 time series increments into the future and past. From there, we calculate the change in the y axis of the derivatives. See example below.

Tangent-calculus.svg

By doing this, we can predict what the change in the derivative is up to 30 minutes before we get to that point. This is key because we’re looking specifically at the slope of an extrapolated derivative. But how do we know when an anomaly may happen? We look at the moving average of the past 14 days of the change in derivative slope. If the current change in slope exceeds the moving average, we’re likely to have an anomaly on our hands. However, we have found this to also be a bit too sensitive by itself which led to creating a combination of both the velocity vector moving average and the derivative slope moving average.

By combining both, we force a decision to be made. If the velocity vector is within the moving average but the derivative slope isn’t, it is most likely not an anomaly. Conversely, the same also applies. What I did find out though was that if both the derivative slope and the velocity vector exceed the moving average, it’s a strong indication that an anomaly is or will happen. I’ve also tried pairing this with 1 standard deviation away from the moving average as a dynamic threshold. Adding this in creates a system that only pulls out the very extreme cases of anomalies. In further tests, I’ll probably be testing out using units of standard deviation as a way to create a “more/less” sensitive alerting system. Almost like a user-drive knob or refinement method.


Is this a finished approach? Absolutely not. There’s a lot of challenges in getting this to work 100% properly to the point that it meets some sort of statistical rigor. I’ve been encouraged by the early results looking at real examples of events happening with our customers in the Twitter sphere. So far, I’ve seen a decent amount of success with predicting when an anomaly may be happening. There are other methods that we could look at to help refine the model, such as adding an F-Score for precision and recall for better accuracy on the prediction front.

Virtual Reality: The Catalyst to Making Money in Many Realities

Posted by | Technology | No Comments

Here’s a fun question.

With virtual reality getting to the point where it hijacks our reality, is it possible to live in one reality but make money in a different “reality”?

There’s a ton of great Youtube videos online showing how immersive virtual reality can be. People fall over, get vertigo on virtual rollercoasters, and get really scared when they turn around to face a fictitious demon. As the hardware starts to push on the software to help drive the innovation, as consumers we’re now reaping the benefits of truly immersive gaming through virtual reality. But this begs the question: how far can we go with this? If individuals are now able to make a livable income streaming their gaming sessions, could we take it a step further? If we’re able to exchange virtual items for real money (Eve Online, World of Warcraft, etc.), aren’t we a few steps away?


Let’s take a step back so that I can tell a somewhat sad story about my childhood. Like many other kids my age, I was obsessed with video games. Specifically MMORPGs. The day I discovered Runescape was the day that a new world opened up to me. I had always played games like Halo and what not but they were a guided gameplay. Scripted. Runescape was different. It was open, had no script to follow, had a new world to explore. Probably most importantly, I could be something totally different than what I could be in my current reality. There was a point where it was more fun to be in the reality of Runescape than my real reality.

As nerdy as that sounds, I knew I wasn’t the only one. In fact, there are millions of people who feel similar. There’s tons of people who get home from their real life, sit down at their computer, and log into a world where the constants of physics are changed, where magic is actually magic, and the limitations of exploration are far more grand than our single planet we live on. It became incredibly appealing to log into a different reality when times were tough. Parents fighting? At least I could escape the yelling. Girlfriend breaks up with me? Log into a vast world of exploration to take my mind off it.

I would say the first time I had the thought about making a living (actual cash) was when I first exchanged a virtual item in Runescape for cash via Paypal back in 2007. While I laughed about it at the time it sort of struck me that the value of something that is virtual is all about perspective. What I felt as totally useless, someone felt it was valuable enough to pay cash for. My hours spent gathering, crafting, and building something virtually was worth actual cash. Said a different way: I was able to exchange my time for cash. Now, how different are these realities again?


Back to today and the virtual economy space is growing. In just the social network space, the virtual currency economy is estimated at roughly $11.3B in 2016. Even with individual game title launches that come with the ability to trade virtual items for cash there is a huge economy driver. While many video games have tried to stay away from the virtual to physical purchasing of items, many have failed. For example, you can purchase characters, gold, items, and many other items for your virtual experience through different forums. There are troves of websites dedicated to this effort, such as Player Auctions.

If we’re willing to purchase items in a 2D experience, what happens when they become close to “3D” with big leaps in virtual reality? I think there’s a couple outcomes that I alluded to in the title of this post. I think that in the next couple decades or so we may see a shift in money making. This isn’t to say that our sole source of income will be done from a virtual world, but rather that there is the potential for a substantial amount of income to be made simply from logging into a different virtual reality. Hence the title of this post. There are a couple of reasons for this: automation of jobs, basic income, and further virtual reality immersion. Let’s walk through each of these.


Automation of Jobs

It’s no secret that software is eating the world. Furthermore, with massive progression being made in software automation, machine learning, artificial intelligence, and robotics, there’s a good chance that many jobs will disappear over the next 50 years (or less!). As humans start to dominate many aspects of life here on Earth through software, we’ll be creating a way for us to free up time. What will we do with all our free time? I think we’ll travel, see a creative boom, and see a big movement into exploring other worlds – virtually. The web could become a place that we actually go to. Experience it. And to further that point, there’s a chance that the next “huge” idea is that developers will engineer new worlds that we can virtually live in for entertainment, for work, for adventure, for whatever. As individuals living in a real world where our jobs have been automated away, we could use our free time to log into many different virtual worlds shared by others. There would be a boom of many different potential realities but, in the end, there would only be a handful of virtual reality worlds that the general population lives in (much like the dominant sites we interact with each other in today). This isn’t too different than how humans currently operate where large populations aggregate into small areas, such as New York city, where the economy is thriving or where opportunity is made.

Basic Income

With the automation of our jobs, what will we do for money? As I mentioned in the last section, there could definitely be a renaissance boom with creatives driving the new world. But if we’ve automated many of our jobs, how will the population at mass make money? I believe this will be accomplished through basic income. Call me a socialist or whatever but I believe that this is inevitable. The other option is to have massive overthrow of the rich and redistribute wealth forcefully. But, for the sake of argument, let’s say that Basic Income is a thing we have in this new future. With Basic Income, it frees us up to not have to worry about the staples we need in our physical reality. We can easily purchase food, get around, and live an easy life here. If basic income happens and people have a lot of free time, I think we’ll see an explosion of a new type of entrepreneurship – a virtual one. If we’ve automated many of the innovations here in current reality, the next obvious step is to dive into a new reality and be an entrepreneur there. That all said, I don’t this Basic Income is a requirement for this to happen however I think it would help usher it in much faster from an adoption rate perspective. Again, we’re talking 30-50+ years from now.

Virtual Reality Immersion

The progression of virtual reality hardware has been quite impressive just in the past couple years. With Oculous Rift being the first the really make the push forward with a product that was promising, many others started to jump on the band wagon – from Google to Microsoft to Sony. Probably the furthest along in both the hardware and software front is Magic Leap. With Magic Leap, they’re marrying the two worlds (base reality and virtual reality) into one optic. These progressions have been primarily focused on the visual aspect of virtual reality however there are big strides being made in deeper full body immersion. The two best examples of this are PrioVR and Virtuix. PrioVR allows you to strap in with different sensors on your body in order to interact with the game. As their demo shows, your TV sits as sort of the main stage with you having the ability to hide behind trees to peek out into the main stage. With Virtuix, you actually stand inside of a component that allows you to run around, “physically” interacting with the game versus on a controller. This, paired with the headset immersion, provides a good glimpse into the kind of full body immersion needed in order to satisfy the potential to “work” in a different reality.


This is really just a theory of one potential outcome for virtual reality. As I said before, I believe that if we get to this point that there will really only be a couple companies that end up creating different realities that are so good that they naturally attract people to log into them. Much like people love New York or Los Angeles because of it’s appeal and their personal taste, I think people will have the same affinity with worlds that share their taste for reality.

To me, I think the really unique concept behind different worlds is that the “makers” of these worlds get to choose the boundaries of the environment (or lack of). Meaning, if we want to change the laws of physics to to allow for things like time warp drives or magic or floating houses, we can. This is especially interesting because it opens up an entirely new way of looking at crafting things. For example, if you look at games like Minecraft, there’s a lot of potential to build economies within the game. The game has it’s own set of rules for crafting different items that require more or less materials, more or less time, and are more or less valuable.

At the end of the day, what I’ve basically been talking about this entire time is really the vision the Wachowski siblings had in their movie “The Matrix” – minus the crazy evil killing machines. Call it crazy or call it far out there. But I actually really start to question it – how far off are we? I don’t envision a world where we’d plug devices into the back of our heads to immerse ourselves. However, I think the concept is definitely interesting and maybe they aren’t that far off from what we’ll be able to experience in the next 30-50 years. With the rate of innovation in virtual reality hardware and software, computing, and video game graphics, is it really that farfetched to think that something like “The Matrix” could happen?

Whose to say we’re not actually in on of many realities? Elon Musk said it best.

Virtual reality is quite transformative. You really feel like you’re there, and then when you come out of it, it feels like reality isn’t real. I think we’ll see less physical movement in the future, as a result of virtual reality stuff, and as the technology improves it becomes, beyond a certain resolution, indistinguishable from reality. There are likely to be millions, maybe billions, of such simulations, so then, what are the odds that we’re actually in base reality? Isn’t it one in billions?

If there’s anything that physics can teach us about reality it’s that if you want to experience it different than your peers, all you need to do is just change your perspective.

Best Product Management Interview Questions I’ve Been Asked

Posted by | Life, Startups, Technology | No Comments

If I’m counting correctly, I’ve interviewed in the ball park of 30-35 different companies for different roles, often times doing multiple rounds of interviews. Of these, around 26 of them have been for product manager roles in varying capacities, from typical product manager roles to director or principle product positions. I’ve been asked some god awful questions (such as stupid challenge/trick questions) to exceptionally great and hard questions. I’ve done extremely well in some and felt great about the outcome, others I’ve felt like shit and totally bombed.

Over the years, these are the questions that have stood out to me the most as great questions to ask at each of the different interview stages. Note that this isn’t meant to be a full comprehensive list, just the ones I personally felt to be the right questions to be asking.

Phone Screening/1st Round

Why our company? What caught your eye? I like this questions specifically because it shows why a candidate is interviewing there in the first place. Are they here for a paycheck or do they actually have some passion about the problem space?

What do you see as a challenge in this industry? Even if the candidate doesn’t have a great grasp on the industry just yet, what you’re looking for here is if they have opinions about what may be hard. For example, a candidate might say that data collection for a certain vertical is hard and follow up with why they believe that. This shows that they did some research, can hold an opinion, and are genuinely curious about the problem space.

In-Person/Onsite

Describe to me a product you love and how you would improve it. For me, the best results of this have been when the candidate whiteboards this out. I find this question interesting because you get to see how they generate solutions on the fly to things they see problematic.

Let’s say you have a Kindle. You buy a book and then read it. Draw what that process looks like. There are no right answers to this question but very wrong ones. This is best captured on a whiteboard. As an interviewer, the best results is to sit back for 5-7 minutes and just let the candidate go at it without any help from you. Be silent. This questions helps show how a candidate views problems, maps them out, and works through different scenarios.

For example, to answer part of the question the candidate would probably think about things such as handling authentication, book delivery, book recommendations, email confirmation, etc. The candidate then may question how exactly do you deliver that book. Where does the book come from? When we deliver it to the Kindle, what other information are we passing down? Where does that information come from? It’s an exercise that shows how someone goes about problem solving, what their logic process is, and where they are strong/weak at.

What do you do in your spare time? What sort of personal projects are you working on? It’s good to know whether or not a candidate has outside interests and if they have a drive to continually learn.

What books are you reading right now? What do you enjoy reading about? More of a personality question but, much like the previous question, this is a good way to understand who the candidate really is. What interests them? Do they like to learn? What areas of personal growth are they focusing on?

Second/Final Rounds

We have this problem in our company. How would you solve it? One of my favorite things to do is whiteboard with people. I enjoy problem solving and find this question to be great for both parties to get a feel for how each other work.

As a candidate, I want to see what you’re like when I challenge your opinions, suggest new ideas, and how we would work together to solve problems. As an interview, I receive the same benefits but also get to see how you problem solve on your feet.

What do you want to be when you grow up? Where do you think you want to end up? Lots of companies look for ambitious product managers who want to learn, grow, and progress their career.

Take a walk with the candidate. This isn’t really a question but more of a personality filter. Get them out of the office, have them relax a little bit, and see what they’re all about. If you’re going to be spending a lot of time with someone, make sure it’s someone who you can be comfortable with.


There isn’t one way to interview candidates and every company or person has their own flavor for figuring out whether or not they fit. Some require presenting a roadmap, others weigh personal history and background.

What questions were you asked that you felt to be the “right ones?”