Guy Nadivi: Welcome, everyone. My name is Guy Nadivi and I’m the host of Intelligent Automation Radio. Our guest on today’s episode is Matt Sanchez, Founder and Chief Technology Officer of CognitiveScale, an enterprise AI software company. CognitiveScale is number one in AI patents among privately held companies and number four overall since 2013, with a focus on helping clients to pair human and machine. Prior to CognitiveScale, Matt was one of the leaders at IBM’s Watson Labs, which is of course, part of IBM Research, one of the largest industrial research organizations in the world. So with such an accomplished track record in the field of artificial intelligence, Matt is someone we absolutely had to have on our show. He’s been kind enough to carve some time out from his understandably busy schedule to join us and share his considerable insights with our audience. Matt, welcome to Intelligent Automation Radio.

Matt Sanchez: Well thanks, Guy. I’m glad to be here and appreciate you taking the time to discuss some of these topics today. It’s certainly a interesting time for us in the field of artificial intelligence, so glad to be here.

Guy Nadivi: So let’s start by talking about something very interesting that you’re an advocate for which is something called responsible AI. Can you please define that, Matt, and explain what the components of responsible AI entail?

Matt Sanchez: Sure. So responsible AI at its core really is about, for us, it’s about wanting our clients to have a better understanding of how to maximize the value of AI while minimizing the risk and the risk could be to their business and it could be to society. And so we believe that you need to have tools to really handle this. It’s not something that many businesses are equipped with today, and these tools need to be able to automatically detect, score and mitigate risks that come from using AI and related technologies to automate decisions or to help augment decisions that are happening in the enterprise. And so we want to make AI transparent, trustworthy, and secure by providing these tools. And responsible AI is really about leveraging those sorts of things to make sure that these systems we’re creating are not just opaque learning machines, but they’re actually trusted, controlled, intelligent systems that can actually improve individuals, organizations, and society. And we think of really there’s six key components to responsible AI. We call them trust factors and we really talk about it in terms of it being a trusted AI framework. And these trust factors are things like effectiveness. So making sure that the AI systems and the models that we build in these systems are continually generating the optimal business value. It’s actually been studied. A number of studies were done recently, even that talked about the difficulty in making sure that AI models deliver ongoing business value. And so that continues to be a challenge, but beyond business value, there are risks that also come with it. And so these trust factors go beyond just understanding business value and look at things like explainability. How explainable are the decisions that these AI systems make to human users? We did a very simple explanation for how an automated decision was made. What about bias and fairness issues? How do we know if there’s bias in these systems? How do we test for it? How do we measure it if it’s somehow hidden or learned, inferred by these AI models? Can we test for that? Robustness, making sure that these systems are secure, that adversarial attacks on these systems can be understood. We can actually test for these weaknesses in these AI systems. Data risks. Data really is the fuel for these systems and if it’s tainted with bad information, right, it’s a garbage in garbage out problem. We need to be able to detect that and data is constantly changing. So this isn’t a one-time thing. It has to be monitored. And then finally compliance. There are legal, ethical considerations when we use automated decisioning. Many countries are actually starting to define very specific laws around automated decisions and the use of algorithms. And so compliance is going to be a continually changing landscape, but one that’s increasingly important for customers using AI.

Guy Nadivi: Matt, you’ve spoken about the need for data to be “nutritious, digestible, and delicious,” end quote, which by the way, is how I like to describe my wife’s cooking. What did you mean by nutritious, digestible and delicious data?

Matt Sanchez: Yeah, so, and I think somebody you had on your podcast in the past, Lee Coulter, is a good friend of ours and someone that we actually have talked about together on this topic, kind of came up with this set of things and tried to create an analogy to what does it mean to really power a artificial intelligence system or to power machine learning? And we were really focusing on the data. So delicious really means the right variety. I need to make sure that the data I have is not, it has the right inputs. It has the right conditions. It has the right outputs. If it doesn’t have that, I know I don’t have complete information. So whatever I’m learning from that, it’s not going to be very good. My results aren’t going to be very good. Digestible means that I have to be able to actually consume the data. A lot of data that was created in enterprises is not digestible by machine learning algorithms. So the structures that are created have to be both useful and usable by the model that we’re creating with it. And it has to be free from any sort of contaminants as well that could cause the system to reject that information. And finally nutritious really means data that really is sustenance for the main purpose of that model. It contributes either positively or negative to the inferences we’re making, but it’s not just noise. It’s not just filler. It actually needs to be the right stuff. And nutritious means that our confidence in the predictions that these systems are making is growing over time. We call that trusted decisions. There’s transparency and trust in those decisions. And then finally nutritious also means that the data itself is not poisonous. There’s no leakage of private data that was unintended or biased information. And so we talk about this as a high level framework to really think about your data because without the right data, AI really cannot succeed.

Guy Nadivi: Matt, as I’m sure you’re aware, AI projects have an unacceptably high failure rate, as much as 85% according to one report. In your experience, what are the biggest reasons AI projects fail and what can be done to reverse poor outcomes?

Matt Sanchez: Well I think there are three key problems for failure in these projects. One is, of course, as I just discussed, is data quality. And if I can’t really get a handle on data quality, then everything else downstream from that fails. You can almost think of AI as a supply chain problem where the upstream work is really around data. And so data quality becomes really key. The second part, which is really one level downstream from data, is modeling and model validation. We’ve heard from clients that it can take upwards of a year to build one machine learning model and get it into production. And the challenge that they tend to get into, maybe half the time, is actually the technical work. The other half of the time is validating that that model is actually trustworthy, that it’s compliant, that it actually delivers the right business value and that we can prove that and that can trip up these projects, essentially stalling them out in the lab. And then finally business outcome, making sure that on a continual basis that my AI systems are measurable against the business KPIs that they’re designed to solve for. That’s the only way to make sure that the investments in those AI systems are actually paying off. And so these three problems really trip up a lot of projects. And what you really need to solve for this is first you need trusted data. You need data that is free from bias, that has the right nutritional value to solve the problem and is ready for machine learning. You need trusted decisions. So we need to make sure that the decisions that these systems are making have a level of transparency and explainability built into them. And then finally we need trusted outcomes. We need to know that and have full transparency from the business side that the AI systems are actually generating value.

Guy Nadivi: Matt, there are, as I’m sure you know, some concerns cropping up about the misuse of AI and machine learning, deep fakes being just one example. Do you see any economic, legal or political headwinds that could slow adoption of these advanced technologies, or is the genie out of the bottle at this point to such an extent that they just can’t be stopped and perhaps not even effectively regulated?

Matt Sanchez: Yeah. So, I think this is a continual challenge in the field of artificial intelligence and in a lot of potentially other related fields. And certainly, there is a lot of opportunities for misuse of these technologies. And I think we will continue to see that by bad actors. Now, that being said, I think most corporations, governments are actually going to, or are incented if you will, to use AI in a responsible manner. And the reason for that is that it’s a brand trust issue and it’s a public safety issue if you’re in a government agency. And brand trust has been shown to be a very costly thing to lose in terms of dollars and cents. And so at the end of the day, if the consumer doesn’t trust the brand, they stop using that brand. That’s results in literally trillions of dollars in lost revenue globally every year because of trust issues. And this can exasperate those trust issues. If you’re using AI in a way that is not trusted, your brand will erode very, very quickly and consumers are keenly aware of this. So that being said, I think to the extent that organizations can address the ethics question around AI, what are your principles as an organization that you’re going to adhere to? Publish those principles and then have a way of actually showing that you’re following them. I think that’s one way organizations can sort of get around the fear, if you will, that they’re somehow using AI for evil in the back office, but also from a regular regulatory standpoint, we’re seeing more and more examples of consumer data protections. That’s usually where it’s starting with things like GDPR in Europe, but now also in California. January 1st this year, the California Consumer Privacy Act went into effect. And now consumers are gaining more control over their own data. And that’s really where it starts and that’s regulation and laws that are being passed. And there are many other legal ramifications for organizations that try to use AI in the wrong way, and particularly try to use data in a sort of illegal way. And then finally, I would say on the public safety side of things, I do think there are genuine public safety concerns with things like autonomous vehicles and other sorts of autonomous technologies that will be regulated at some point. We will have to expand the regulations that exist to protect the public from these technologies, just like any new technology that surfaces in the marketplace. And then of course, there’s always the bad actors who try to use technology for their own criminal purposes. And that’s going to happen whether business uses AI or not. In that sense the genie is out already out of the bottle. The technology is out there.

Guy Nadivi: Your company CognitiveScale describes its product as, “the world’s first automated scanner for black box AI models that detects and scores vulnerabilities in most types of machine learning and statistical models.”. There are some who say that AI algorithms should be audited much like a publicly traded company’s financial statements. Could CognitiveScale be a virtual operator for auditing AI algorithms?

Matt Sanchez: Yeah, that’s a great question and, in fact, one that we talk to our customers about quite often, and as it turns out, auditing is already starting to occur. There are large organizations that have had various forms of, I’ll call them, AI audits and particularly auditors starting to look into from a business risk standpoint. How has the use of AI potentially introducing new risks into the organization and are they managing those risks appropriately? You can think of banks, for example, needing to answer these questions from a regulatory standpoint. And so AI auditing, if you will, is becoming an increasingly important topic. Now how do you do it? First of all, I think you have to understand the ethical principles, regulations, laws, et cetera, that are applicable for your business and for your jurisdictions of interest. Regulations are specific to jurisdictions. In the United States, for example, if you’re a healthcare insurance company and you operate in 50 States, you probably have 50 different sets of insurance codes that talk about discrimination and they don’t all talk about discrimination in the same way. And so now you have to understand how to apply what does that mean for your business? And a lot of organizations, what they’re doing to get ahead of that is to define their own and publish their own AI principles. You can see large tech companies like Google, Facebook, and others who have published some of these principles, but now you can actually start to see banks and healthcare insurance companies and other types of companies starting to publish their AI principles. What are their values as a company and how are they using AI responsibly? And so that’s sort of the second part. First understand what are the applicable regulations and laws? Second is define your principles. And third, have the measurements and controls in place to prove that you’re being compliant with these rules and regulations. And on that last point, this is where our product, Cortex Certified, can really help. Because one of the things we discovered is that within an organization, the technical people, the data scientists and the engineers, speak a very different language than the compliance officers and the business owners when it comes to AI. And so we needed a common language so that they could all talk. And this is something we call the AI trust index. Think of it as almost a single scoring mechanism for measuring algorithmic risk and breaking it down into those six trust factors that I discussed earlier, where we can now get a very simple score, almost like a credit score for an AI that tells me how trustworthy is it. And so instead of just looking at the technical attributes, the statistical attributes of these systems, I now can look at the trust attributes or the ethical attributes of these systems. And this is enabling a common language to be able to then facilitate measurement and ultimately controls and audit in these organizations.

Guy Nadivi: Interesting. Earlier this year, Matt, there was an article in MIT Technology Review about artificial general intelligence or AGI. In that piece, the author, Karen Hao, who’s been on her podcast, wrote quote, “There are two prevailing technical theories about what it will take to reach AGI. In one, all the necessary techniques already exist. It’s just a matter of figuring out how to scale and assemble them. In the other, there needs to be an entirely new paradigm. Deep learning, the current dominant technique in AI, won’t be enough”. Matt, what do you and your team at CognitiveScale think it will take to achieve AGI?

Matt Sanchez: Yeah, so I’ll preface this with the point that our team at CognitiveScale, AGI is interesting and it’s a topic that’s worth debating. However, my view is this is not where the current opportunity in the market is. And so while it’s interesting, we don’t spend a whole lot of time in the AGI world, but that being said, I do have some opinions that I can share. And I think there’s a couple of different ways of looking at it. One is if AGI is supposed to be creating a system, creating technology that can really work like the human brain, meaning think and learn the way that the human brain does, then we have a long way to go, like maybe 50 plus years of more work to do before we even get close. And I just pointed to two things that humans do that machines don’t do today at all and we don’t even know how to do at the scale that the human brain can. And the first is just common sense, understanding or common sense reasoning. And this was something that I learned about when I was at IBM, because we were certainly trying to figure out how to teach Watson to have a little bit more common sense when it was answering questions and it’s challenging. It’s challenging to some of the things we learn as human beings, the inflections in our voice, the subtleties of body language, things that just are intuition to humans are very difficult for machines to understand, and encoding that information, encoding that data in a way that the machines can understand is really challenging. So, in that sense, I would agree with the person who said that we need new technologies to solve for this because the encoding of that is still a big challenge, even with deep neural networks. And then things like emotion are also very challenging and they factor very deeply into how the human brain works. So if that’s the definition of AGI, I think we’ve got a long way to go. If AGI really at an algorithmic level is supposed to be about generalization, so generalizing, showing that one algorithm can solve for multiple tasks, different types of tasks without having to be explicitly retrained, if you will, or rebuilt to solve those tasks, then I think we’re actually on our way. I think there’s been some great advances along this dimension with reinforcement learning and some other technologies. And so there are certainly a lot of interesting advances in this space, but my view is, the definition of AGI that I’ve always sort of looked at really talks about it being more of this learning, really simulating, understanding and learning in a way that’s similar to how the human brain can reason and learn and generalize. And I think we’ve got a long way to go before we are even close to that.

Guy Nadivi: Okay. So perhaps that’s a good segue into my next question. Overall, Matt, given your vantage point, what makes you most optimistic about AI and machine learning?

Matt Sanchez: So the number one thing that I get excited about with AI is when I see real business outcomes. So when I see that by using AI, we can start to save a lot of money for our customers, or we can help them improve the customer experience that they want to deliver. And when I can see that in terms of dollars and cents, for me, it shows that AI is working, that it’s worth the investment and it actually is something that’s worth pursuing as a core capability in the business. And so I think that’s, at the end of the day, what I get excited about, and we see that with a lot of our efforts and with our customers. And in fact, we make that a core part of our methodology for how we work with clients is to really focus on the outcome first and to really challenge our clients and ourselves to define what that outcome is, how we achieve it. What does good look like? Why is it better than what you’re doing today? And I think if you start from there and when you see the result and you can calculate the value, it’s really exciting. And I’ve seen so many examples of that now over the years that I’m really excited about the future. I think we can continue to improve and to apply that. And it’s really this iterative process where the first time you turn the crank and see this result, it’s very exciting and it makes you want to do it again and again, and, as you do that, your data gets better. Your techniques get better, your infrastructure gets better, and you start to see things go faster and faster. And we’re seeing examples of that today with a lot of our clients that are making it that I’m really excited about. The second thing I’m really optimistic about is that I think ethical AI and the concerns around this that are really top of mind, both for consumers, for governments, perhaps not as much the US government, although recently there’s been more movement there, but in other countries, Canada, Europe, even the Middle East, there’s a lot of proactive effort from the government side to really define the principles again at a societal level around AI. And I think that’s really encouraging because to me it means that people are starting to understand how to define this and that it’s important. And I’m also seeing it at the level of CEOs and boards within very large corporations because again, they’re worried about risk and they’re worried about the brand trust. And they know that AI has both the potential and the power to be very valuable, but also very, very risky if they aren’t managing these things. So I’ve seen an increase in that dimension. And I think that you can point to a few events that have occurred, very public events that have occurred, that you can kind of see why these issues are top of mind and things like data breaches, data misuse by certain organizations and social media outlets and people being put in front of Congress to explain themselves. I mean, these are things that no CEO or board wants to be a part of. So a lot of these challenges now have been recognized and organizations are starting to invest in making sure that they can get the right outcomes from these systems and do it in a safe way. So that’s why I’m excited about it. I’m seeing that trend increase, both of those trends increase.

Guy Nadivi: That is all encouraging. Nevertheless, as a corollary to my last question, I’ve also got to ask what leaves you most concerned about AI and machine learning?

Matt Sanchez: Yeah. So a couple of things. One, I would say the first one is inflated expectations. AI is not a magic wand to solve all your past data sins is what I like to tell people. So back to my comment on garbage in, garbage out, if your data is not nutritious and digestible, AI is not going to solve that for you magically. And it really comes down to the information you believe you have is subjective. A simple example of this would be if I’m trying to solve an image classification problem, I want the machine to tell me if the image I’m looking at is one thing or another. If I put two images in front of two different human experts and those two different human experts disagree on the classification, then effectively what we have is highly subjective information. And it’s likely that AI is not going to provide a whole lot of value there. It might. AI could provide some additional information that could help those human experts, but it probably isn’t a situation where it can automate that decision in place of human intervention. So we have to always think of AI, it’s not a magic wand, but it can help. It can be, it can certainly help. It can certainly potentially uncover some of those ambiguities and actually improve upon them and make your data better and make your processes better. But inflated expectations is sort of the one thing that always has me worried. Second thing is over-hyped fears. As we said, I think we do have the ability to put the right guardrails around AI. We do have the ability to measure things like explainability and robustness and bias. And I think corporations will do this because it is a brand trust issue. It is a legal compliance issue, but the fears that the corporations are going to start using AI to somehow abuse people’s information, they’re real situations and a lot of times it’s unintended side effects. And I think that’s the challenge. I think that’s what we need to really focus on is not necessarily that it’s ill intent, although that does happen, of course. There are always bad actors and I think we all hope they are the exception and not the rule, but be given that there is a way to measure these things, I think we have the ability to actually put the guardrails around these systems. And I think that’s important for us to work with leaders, leaders in the government and business leaders to really make sure that those practices, those controls are put in place. But those are the things that worry me the most, that the expectations are inflated and that fears are also in somewhat over-hyped in a science fiction type of way sometimes and in other ways. There are some real fears. There are some real issues that have occurred, both bias related issues, where it’s almost we like to think of it as this fairness through unawareness fallacy, which basically says, “Well, of course I’m making a fair decision because the system doesn’t even understand the concept like gender or age or ethnicity.” But we actually can prove is that sometimes those systems, even though you’re not explicitly telling them that information, because it’s the way that machine learning works, they sometimes learn those patterns and can develop biases that you don’t want. And so the idea that you can just be unaware of these things, and that makes you fair is actually false. And so I think it’s those types of understandings that can overcome some of those fears.

Guy Nadivi: Matt, for the CIOs, CTOs and other IT executives listening in, what is the one big must have piece of advice you’d like them to take away from our discussion with regards to deploying AI & machine learning at their organization?

Matt Sanchez: Yeah. So really I would kind of say it like this. Start with the business outcome in mind. Set realistic expectations with the business around what that outcome’s going to look like. Make sure you add explainability and measurement as a first class requirement to these systems. So with the business outcome in mind, also ask how are we going to measure it? Do we have the right feedback loop in the system to really measure this? Make that a requirement of the system, not an afterthought. And be prepared to iterate, to deliver incremental value, meaning you’re not going to get it right the first time. You’re going to have to iterate. You’re going to learn a tremendous amount every time you turn the crank on these systems and they do improve over time. We’d like to say that with AI the first day is the worst day, meaning the very first version of your system probably is the worst it’s ever going to be. And this is somewhat unique about AI systems. They improve with time. They improve with that feedback loop being put into operation. And that’s very unique in the IT world because most of the IT systems we build, they realize their maximum value on day one and it sort of then declines over time. AI kind of works the opposite way or it should. And so a big part of that is iterate. Think of it as an iterative process. Start small, and stair-step your way towards incremental business value.

Guy Nadivi: All right. Looks like that’s all the time we have for on this episode of Intelligent Automation Radio. Matt, it’s been a real treat having a marquee name in the field of AI on the podcast today. I think you’ve really shed some light for our listeners on the black box that is artificial intelligence. And I suspect you provided many of them with new data points to factor into their thinking for their own AI projects. Thank you very much for being on the show today.

Matt Sanchez: Well, thank you, Guy, and appreciate the time and look forward to hearing the podcast when it goes live and following other topics in the space that you’re interested in.

Guy Nadivi: Matt Sanchez, Founder and Chief Technology Officer of CognitiveScale, an Austin, Texas company. Thank you for listening, everyone. And remember, don’t hesitate, automate.

Guy Nadivi: Welcome, everyone. My name is Guy Nadivi, and I’m the host of Intelligent Automation Radio. Our guest on today’s episode is John Gentry, Chief Technology Officer and Senior Vice President at Virtana, a leading AIOps platform for digital transformation. The job of monitoring and managing IT operations continues to grow progressively more complex and increasingly more mission-critical as enterprises move more of their processes to the digital realm. And AIOps is the solution many senior IT professionals are counting on to help their organizations keep up. But can the application of AI, machine learning, data science, automation, and other technologies provide IT with what it needs? John Gentry is a forthright advocate for the innovation and positive disruption AIOps can bring to bear on enterprise infrastructures. So we’ve brought him onto the show today to help our audience understand how real AIOps is today and what it’s likely prospects are for the future. John, welcome to Intelligent Automation Radio.

John Gentry: Thank you very much, Guy. I appreciate you having me. It’s a pleasure to be here and I really look forward to sharing at least my experience with customers and the state of AIOps in the real world.

Guy Nadivi: And that’s exactly what we’re about to dive into. AIOps is very much a market in rapid evolution. Even the acronym, AIOps, itself represents something different today than it did when a Gartner analyst named Colin Fletcher coined it in 2016 and originally Colin meant the term to refer to Algorithmic IT Operations. But since then, it’s evolved to refer to Artificial Intelligence for IT Operations. John, for the people in our audience struggling to synthesize all the marketing messages they hear about AIOps, how do you define what AIOps actually does?

John Gentry: That’s a great question and certainly something that needs clarification and will probably continue to need clarification and refinement for quite some time. I know that it’s even evolved since the rebrand to artificial intelligence where now they’re categorizing two subcategories. One called General Purpose AIOps, which is applying data science and AI and machine learning to general sets of data for things like pattern recognition, and anomaly detection, maybe deduplication. And then they’ve entered a new term called Domain-Specific AIOps, which is really applied to just that, a specific domain, whether that’s a certain component of infrastructure operations or the application of AI to application performance monitoring. So I think it’s going to be continuing to refine, I think filtering out the noise. I often say to customers asking questions around how to effectively leverage it, that they peel back the onion. There’s a lot of marketing claiming that there’s artificial intelligence or machine learning, two very distinct things, being applied to their platforms. And if you look under the covers, there may be some fairly simple statistical regression or pattern matching, but there’s really no deep machine learning or other applications of advanced science. So when I look at how you define AIOps and what it actually does, I think you need to look at applying very specific mathematical or data science approaches to very specific sets of data that have context to reach purposeful or predictive outcomes. And so that can really introduce a wide spectrum, but I definitely recommend peel the onion back, look for the true application of various forms of data science to very specific sets of data to solve very defined problems.

Guy Nadivi: Okay, that’s the kind of specificity the market needs. Now speaking of the market, there have been some high-profile transactions in the AIOps space this year with ServiceNow acquiring Loom Systems, LogicMonitor acquiring Unomaly, and VMware acquiring Nyansa. Virtana has made a few acquisitions itself, but still remains an independent AIOps vendor. If an organization is considering AIOps for their environment, why go with an independent vendor rather than a larger software vendor who includes AIOps as part of a suite of offerings?

John Gentry: That’s again a very good question because there are definite trade-offs in the two approaches or in the selection of one versus the other. In some cases it may be a matter of actually leveraging both. When you look at some of these acquisitions, I think there was a realization in some of the traditional approaches that companies like ServiceNow or VMware have taken historically to IT operations that were organization of information and visualization of information. And they were potentially behind the curve in terms of natively developing AI as an intrinsic part of that process. Something unique about Virtana, we recognized because of the sheer scale of the data that we collect, the need to start applying data science and applying analytics back in the 2013 timeframe. Several years of experience, not only with understanding the data, but with applying the right data science to get to a specific outcome, I think those big providers, they’re going to be challenged to leverage or layer in the intellectual property they have acquired effectively. That would be a caution I would put around looking to a more of a generic solution provider that’s embedded AIOps versus an independent. At the same time, they do have one aspect of the approach that I fundamentally agree with, which is AIOps is one part of a larger puzzle in terms of driving towards things you mentioned in the opening like automation and automation & governance. So, no, I think even your independents need to start considering AIOps as one capability that they bring to bear, but not the definition of everything that they do. I think AIOps is really one aspect of the overall sort of continuum of IT operations when you start to look at everything from the rise of DevOps and the blurring of the lines between App Dev and Operations. And certainly when you look at the rise of public and now what’s presumably going to be hybrid cloud environments, and where can you effectively leverage AIOps to help with decision making around what’s the right place to place workloads? That’s a very different application typically of things like Monte Carlo simulation or advanced modeling versus AIOps with statistical regression analysis or pattern matching to find recurring problems and predict them and prevent them. I think you’ve really, going back to the first question, got to understand what am I solving for, and then am I best solving for that by looking at an additional capability of an existing vendor, like a ServiceNow or VMware, or do I need to find something more purpose-built for the specific challenge I’m trying to overcome, be that problem avoidance in a mission critical system or hybrid cloud transformation in a more bleeding edge sense.

Guy Nadivi: At the time of this podcast recording, the world finds itself in the midst of a global health emergency due to the coronavirus. So John, I think a lot of our listeners would be very interested to hear what contributing role can AIOps play in the current COVID-19 pandemic to help facilitate the sudden surge in people working from home?

John Gentry: Well, that’s one of the things I can say with some certainty is that certainly our solution, which understands seasonality and trends for customers like online trading and transaction processing platforms, most of what is normal, according to the machine learning, is no longer normal, right? So I think one of the easiest or most immediate applications was very quickly identifying just how significant a deviation from normal, the changing in work patterns actually represented, and being able to adjust or accommodate more quickly. I can tell you from personal experience, going back to that online trading company, the fact that they had years of leveraging AIOps and the application to understanding that the traffic patterns associated with things like market open and market close and aftermarket for trading and futures analysis meant that when those patterns all changed, they could very quickly identify where they needed to shift resources to accommodate new and different patterns. We actually reached out to them early in the pandemic when the markets were going significantly haywire, just to make sure they had everything under control. And they actually commented back that it was because of Virtana and the systems that were under management through that solution, that they weren’t having issues in that particular aspect of their business. Now, there were issues in other aspects just because it’s hard to manage everything at once, but that was encouraging to see them, real world application of what they knew about their business to accommodate very quickly identified changes. At the same time we had another customer that if you can believe it or not, a very traditional financial institution, was 100% in-office workforce. They had to very quickly transition to 100% remote. Being a financial institution they had to do so with as little disruption to their customers as possible. They actually used their understanding of predicted capacity forecasting to reallocate resources from traditional applications that were quite frankly no longer being used in any kind of meaningful way, from a volume perspective, and immediately redeploy those resources to support a VDI environment and scale that VDI environment to get their workforce back online. Actually managing the data center capacity or compute and storage capacity to deliver on that transformation was the easy part in their opinion. The harder part was the sheer shipping and logistics of getting all their employees set up with laptops or desktops and monitors and phone lines, but they managed to do it. Thousands of employees went from in the office to fully remote in 10 working days, which I think is a testament to the efficiency of their organization. Certainly it’s a lot more than just IT or AIOps. It’s a lot of really good people in process, but two very different examples. One ability to accommodate massive change by understanding it. The other, the ability to reallocate resources to new applications because of the level of visibility they had ongoing in their environment.

Guy Nadivi: Okay, let’s dive a bit deeper on specific results of AIOps. Can you provide some details on two or three of the most successful AIOps use cases you’ve been involved with? And if possible quantify the impact they’ve had.

John Gentry: Absolutely. I’ll actually talk about two specific ones that are really very different in terms of the business impact, in the very nature of moving or the promise of moving from traditionally reactive to ultimately proactive and even predictive operations. One is a fairly significant customer. They’ve been with us over nine years and their online transaction processing business, in that nine years, they’ve seen their transaction volume grow by somewhere on the order of 700%. You can imagine going from 24 million transactions or 24 million users to 400 million users over the course of nearly a decade. They’ve leveraged AIOps to understand how to gain efficiencies in both the infrastructure and the operations supporting that growth. And so while they’ve seen a 7x increase in their transaction volume, they’ve only had to scale their infrastructure by about 3x, right? So that’s just eking every bit of performance and capacity out of what they’re building to support their business. And that’s absolutely the result of the application of AIOps to understand those business patterns and cyclical patterns that mean I can move resources around versus just building excess capacity to accommodate a burst. And at the same time, they’ve been able to do that and maintain that growth with the same staff. They have the same number of people managing what is a 3x infrastructure and a 7x transaction volume as they did nine plus years ago. So when you think about their business, the ability to transact at a higher volume with less infrastructure, and to do so at ever increasing efficiency per head count, those both go directly to the bottom line. They are directly to the margin associated with every transaction. If it costs me less than infrastructure and it costs me less in people per transaction, that means that I’m affecting the bottom line from a cost perspective and driving profitability. So that’s one area, it’s like gleaning those efficiencies that promise of AIOps to automate the mundane and optimize resource utilization and move from reactive to proactive problem prevention so you don’t need a bunch of head count doing firefighting. And so that’s one example. Another example is actually a more recent customer, also a SaaS provider. Interestingly enough, we seem to have a strong market presence in software as a service likely because the software is the business and the infrastructure to support it is mission critical. But in this particular case, it was leveraging AIOps to drive competitive advantage and actually drive time to market. So this particular customer, they make their money through new feature, new capability introduction, right? They’re a large real estate property management firm. They’re the leader in SaaS-based property management. And when they introduce a new capability, like say risk assessment based on geography, or rent price elasticity analysis based on economics. That’s a new service that their customers can subscribe to and glean benefit from. What they actually did as a company, they are using Virtana in conjunction with one of our partners in the APM space to really drive a single point of view from early phases of application development all the way through production deployment to ongoing sustaining. And what it’s allowed them to do is one, eliminate any finger pointing between App Dev and IT Ops to say, “Well, you wrote bad code.” “No, you got bad infrastructure.” And say, “No, actually, we know exactly where the efficiencies are between the two,” but more importantly what they realized was they could actually build infrastructure that was more efficient at supporting the code, right? Or they could actually write code that better leveraged the underlying infrastructure. I think about my background in storage, moved to all flash, and the fact that all flash is really, really good at processing the very specific block size in terms of a request. And if you tune your query from your database to actually request that block size, you’re going to get a lot more performance for a lot less capacity in that environment. And so what they were able to do, was drive efficiencies between code and infrastructure, and that actually accelerated cycle time from Dev to DevOps, Dev Tests, to actually production release, to go live. And so if they can now take what used to be a 12 week process for building and releasing a feature and cut that to six weeks, they just gained six weeks of market time for that to be in market. That’s competitive advantage from a time to market. It’s competitive advantage from a differentiation and at the end of the day, because they’re doing it with more highly optimized code to infrastructure relationship, it’s competitive advantage in terms of profitability as well. I’m seeing applications of AIOps, we’ve done in a sophisticated manner. And again, it takes people in process that are behind it to get to those outcomes, but I’ve seen it drive cost down and profit up as well as really convert or transform that IT organization to a competitive advantage and differentiator for the business.

Guy Nadivi: John, one criticism I’ve heard about AIOps is that it tends to lack a risk analysis capability. In other words, it can tell you when an incident or issue needs your attention, and it can suggest to you what kind of attention that incident or issue needs, but it doesn’t provide a human operator any background about the risk of taking any of those suggested actions, perhaps with some historical context added about what broke the last time one of those particular actions was executed. One Gartner analyst told me that lots of AIOps vendors are claiming they can do this, but it only seems to work in the operating system called PowerPoint. How soon do you think we’ll start seeing risk analysis capabilities in AIOps?

John Gentry: That is a foundational question from my perspective, certainly given that the nature of our customer base being really blue chip and mission critical environments. I think I would answer that in a couple of ways. I mentioned before that reliance not just on technology, but on good people & process. And so I think there is an approach that I know I recommend and discuss with certainly my more conservative customers, this concept of automation with governance, right? And I think first you have to draw a distinction between AIOps as a learning problem identification and remediation platform where I can identify a problem. If I have a known resolution, I can make a recommendation and separating that from automating taking that action. Right? And I look at AIOps as being sort of the intelligence behind automation, the decision support, if you will. But then to introduce that change through a typical change management or typical governance frameworks. You mentioned ServiceNow earlier. That’s an obvious choice to say I’m going to identify the issue proactively and make a recommendation, push that recommendation along with the underlying root cause analysis to a change management request. And then I am going to introduce that human operator, that judgment call to approve that, to do a manual risk assessment and then execute. I think the next step from there, and certainly where we are in that journey, is having the closed loop visibility to say once I’ve executed that and I have an outcome that I’ve observed, and that outcome was as anticipated or what was predicted in the recommendation engine, as once I do that X number of times, pick your level of risk tolerance, go ahead and automate it, but still introduce or note the change in my change management system so I have an audit trail if for some reason the first 10 times it went great and the 11th it didn’t. That’s really where that machine learning underneath the surface becomes important. So I think AIOps or automation with governance is key. And I think it is going to take an ecosystem, not just a single provider, because again that domain specific knowledge is critical. I’m not going to make certain recommendations around things like firewall settings, certainly, because I’m not a security expert and my platform knows nothing about security, right? So I think that we really want to take a measured look. In terms of when it gets into actual product offerings, I do think it’s mostly PowerPoint today. I think it’s again, an application of AI in a certain environment. The one distinction I’ll make is starting to introduce something called … that is very much about measured risk. And so you mentioned that we had made some acquisitions earlier. Actually it was late last year, and we acquired a company called Metricly and that’s now our cloud wisdom offering, and Metricly, one of their core pieces of IP that was intriguing to us was their very advanced anomaly detection and the way in which they had leveraged that along with very specific heuristics to introduce the ability to manage risk. And so Cloud Wisdom is a SaaS-based cloud monitoring tool. It’s really an acquisition to take us to hybrid cloud and cloud cost management because our customer said one, my cloud costs are out of control, but two, I can’t just slash and burn. I need to understand the impacts on things like performance and availability or risk tolerance of my business. And that was one of the unique aspects that they actually introduced in the product, which was the ability to dial in very specific thresholds for how much performance is required and how much cost is important to me and how much risk tolerance do I have around this particular workload. And then run that through advanced heuristics to say here’s the optimal balance of reserved capacity versus on demand capacity to meet the optimal point of maximizing or optimizing performance cost and risk. And that was one of the first applications I had seen really calling out risk as a variable or risk as an input in those decisions. So it is starting to come. We’re rolling that capability into our broader hybrid cloud management offering, trying to bring that same type of analysis back on-prem, and then let customers manage the balance between what should live on-prem, what can live in the public cloud, and how might I arbitrate that workload placement going forward with a keen understanding of risk. I mean, we have one customer is a very large SaaS provider and they said flat out in looking at the platform, “We’re going to dial the risk tolerance all the way to the tightest. We have zero tolerance for risk and we’re willing to pay a little extra or even a lot extra as a result.” So they actually appreciated that they could mitigate risk by dialing that knob in a certain direction versus another customer that’s only looking at cloud for noncritical systems and said, “I’m willing to make the risk factor a nine and save as much money as possible in the process.” So I do think it’s starting to enter the conversation, but I do think Gartner actually had this one fairly right in terms of its maturity. It’s mostly in concept with a few real world applications.

Guy Nadivi: Well, let’s discuss a different type of risk. The entire value proposition of AI and machine learning being able to get you to the point where you can predict failures before they happen and mitigate them in advance is heavily predicated on one particular skill, data science. If you don’t have the data scientist to build the algorithms to generate the predictions, then you can’t leverage AIOps. And right now there is a very big shortage of data scientists. The August 2018 LinkedIn workforce report stated that there was a nationwide deficit of over 150,000 data scientists. How will companies like Virtana overcome this staggering talent shortage?

John Gentry: Oh, it is absolutely challenging. I think it brings up an interesting sort of trade-off that I’m certainly seeing in my customer base. We have a little bit of advantage because we did have a head start. We started building out that capability before there was a massive shortage. We were a little bit ahead of the curve. I’ve seen customers trying to balance the decision of build versus buy, right? There were a lot of companies that were naturally in the data science business. Think about insurance firms that have large teams doing multivariate analysis for risk assessment and setting policy pricing and so forth. I saw a lot of them early on saying, “Well, we’re just going to pump everything into a massive data lake and apply our data scientists to it. And we’ll build a solution that will solve world hunger in our data centers, but we’ll do it on our own.” I think that is less and less of a reality because the thing that’s lost in just genericizing data science is there’s a level of applied knowledge that has to exist for the data scientist to actually pick the right math and know what outcome they’re solving for algorithmically. And so I think you’re going to see more companies looking to buy the capabilities out of the box as opposed to build them themselves, which should help to some extent that data scientists be able to be leveraged. If I can build a capability into a product and multiple entities can leverage that product, I don’t need to have a data scientist and all those entities. Now that’s really the trade-off between the customer and the vendor in terms of the build versus buy. In terms of the competition in the vendor ecosystem for those data scientists, it’s forcing people to look very hard at geography. Interestingly enough, we made the very conscious decision to open a development office in Bend, Oregon, because frankly, the competition for talent in the Valley is extremely fierce. A lot of the next generation of data scientists, they want to be in a beautiful part of the world that’s not overly urban, that offers them lifestyle choices that allow them to do more than just their job. And we’ve had great success with that location and actually really happy employees there that tend to write really good code. I think you’re going to see that kind of creative sourcing and creative appeal. Not just money and marquis name of company, but also whole lifestyle offerings that attract those data scientists. And then you’re going to see hopefully a lot more students and a lot more diversity in the workforce going into the field of data science, and maybe even certainly some reskilling of the labor force. I think this is such a wave of transformation and the promise is significant enough that I think you’re going to see some retooling to fill that gap. Not just new grads or existing statisticians, but it’s certainly a challenge. It’s forced us to change our organizational structure and our geo-locality. I’m sure it’s having similar effect on other organizations. That’s definitely something I know I’m pushing my son to consider a career in data science. That’s for certain.

Guy Nadivi: I think Bend, Oregon, also happens to be the location of the world’s last Blockbuster video. So maybe that might be another big draw for your recruiting efforts down there. John, I’m going to ask you to fire up your crystal ball and speculate about what kinds of features and benefits AIOps tools will provide three to five years from now.

John Gentry: Yeah, well in the state of the world right now and the rate of change that’s happening, I’m pretty guaranteed to get this at least mostly wrong, but I’ll certainly give it a shot. I think from a features perspective, the promise of AIOps has always been automate the mundane, the repetitive, the painful parts of IT operations to free up the people, the intelligence, the creativity to go innovate. And so I think you’ll see tighter coupling of the AI engine and the automation platforms and more closed loop approaches that can continue to self-learn or self-heal. I know we’ve talked about the self-healing data center. I think as you look three to five years out, they’re going to have to natively understand it is a hybrid cloud world. I think just applying AIOps to a private cloud data center, or just leveraging it for public cloud management is going to fall short of customer requirements to really manage workloads on-prem, off-prem, and transitory between the two locations. If I can proactively understand, based on business patterns, when it’s more cost efficient to run a massive say business closed process on-prem, but during normal operations that’s better done in the cloud, I’m going to want to be able to automate that migration and that arbitration between workload placement between the two. So I think an inherent understanding of the trade-offs between on-prem and public cloud is going to be something that has to be there in the next three to five years. I think increasingly, as I mentioned before, cost is going to be an inherent input to that. I think the promise of cloud being cheaper has been fully debunked. And so now we really need to take a hard look at cost management and how that plays into the arbitration between workload placement. I think more and more simulation test capabilities. AIOps today is very much a reactive type of platform. I think leveraging more advanced mathematics, things like I mentioned earlier, like Monte Carlo simulation to look at all the permutations of a configuration or workload placement to pick the optimal one based on other specific set of inputs, and then to go and test that theory to simulate that, and then validate before taking action. I think those are all capabilities that are going to be required, particularly as you start to move more and more toward humanless automation, right? So if I can actually now leverage AI to understand a pattern, to make a proactive recommendation around workload placement, and then actually go simulate that placement and validate the result, validate my AI, and then execute based on validation and then close loop monitor the outcome, now that’s a very sophisticated full stack, full life cycle approach that I think that’s going to be absolutely what you’re seeing in the winners out there and their capabilities in the next three to five years.

Guy Nadivi: Interesting. John, for the CIOs, CTOs, and other IT executives listening in, what is the one big must-have piece of advice you’d like them to take away from our discussion with regards to implementing AIOps at their enterprise?

John Gentry: Well, I guess the first thing I would say would be don’t believe the hype. A lot of the marketing sounds the same. There’s an incredible perceived overlap in various providers and their capabilities. There’s a lot of promises that for better or worse in IT, we’re used to vendors over promising and under delivering. And there’s that acceptability rate of somewhere between 70 and 80% actually being delivered with AI. That’s not an acceptable ratio, right? Because now you’re talking about really automating and applying intelligence and such to your business. So really peel back the onion, get to the meat, ask the hard questions about exactly what data science is being applied and when and how. I was talking with my CEO about actually building a Hobbesian pyramid of AIOps to explain how the various mathematical approaches need to be layered on each other to get the various incremental returns. So I think really digging into the true capabilities beyond the marketing would be the big piece of advice. If I had a secondary piece of advice it would be really look at what problem you’re trying to solve. One of the most powerful questions I’ve had throughout my career, one of my early mentors infused within me is always ask what are you solving for? I think people get enamored by technology and they go out and throw technology at a problem that may not be the problem that needs to be solved. So, I would say, don’t believe the hype, do your homework, but also sit back and think hard about what is it you’re solving for, and then look for the approaches or the technologies, or even the vendors that seem to resonate in that being the outcome that they’re driving for as well.

Guy Nadivi: Excellent advice. Alright, looks like that’s all the time we have for on this episode of Intelligent Automation Radio. John, your reputation as a fervent and articulate advocate for AIOps preceded you, and you definitely did not disappoint. I’ve really enjoyed absorbing your boots on the ground perspective today on the state of the market. Thank you very much for coming onto the podcast and sharing your views with us.

John Gentry: Well, always a pleasure, Guy. Really enjoyed your questions. Very insightful. They helped get to the root of the issue. So I’m very hopeful that your audience has benefited from this, and I look forward to collaborating with you again in the future.

Guy Nadivi: John Gentry, Chief Technology Officer and Senior Vice President at Virtana. Thank you for listening, everyone. And remember, don’t hesitate, automate.