IT Incidents: From Alert to Remediation in 15 seconds [Webinar Recap]

/in , , , /by

Author: Guy Nadivi

Remediating IT incidents in just seconds after receiving an alert isn’t just a good performance goal to strive for. Rapid remediation might also be critical to reducing and even mitigating downtime. That’s important, because the cost of downtime to an enterprise can be scary. Even scarier though is what can happen to people’s jobs if they’re found to be responsible for failing to prevent the incidents that resulted in those downtimes.

So let’s talk a bit about how automation can help you avoid situations that imperil your organization, and possibly your career.

Mean Time to Resolution (MTTR) is a foundational KPI for just about every organization. If someone asked you “On average, how long does it take your organization to remediate IT Incidents after an alert?” what would your answer be from the choices below?

  • Less than 5 minutes
  • 5 – 15 minutes
  • As much as an hour
  • More than an hour

In an informal poll during a webinar, here’s how our audience responded:

More than half said that, on average, it takes them more than an hour to remediate IT incidents after an alert. That’s in line with research by MetricNet, a provider of benchmarks, performance metrics, scorecards and business data to Information Technology and Call Center Professionals.

Their global benchmarking database shows that the average incident MTTR is 8.40 business hours, but ranges widely, from a high of 33.67 hours to a low of 0.67 hours (shown below in the little tabular inset to the right). This wide variation is driven by several factors including ticket backlog, user population density, and the complexity of tickets handled.

Your mileage may vary, but obviously, it’s taking most organizations far longer than 15 seconds to remediate their incidents.

If that incident needing remediation involves a server outage, then the longer it takes to bring the server back up, the more it’s going to cost the organization.

Statista recently calculated the cost of enterprise server downtime, and what they found makes the phrase “time is money” seem like an understatement. According to Statista’s research, 60% of organizations worldwide reported that the average cost PER HOUR of enterprise server downtime was anywhere from $301,000 to $2 million!

With server downtime being so expensive, Gartner has some interesting data points to share on that issue (ID G00377088 – April 9, 2019).

First off, they report receiving over 650 client inquires between 2017 and 2019 on this topic, and we’re still not done with 2019. So clearly this is a topic that’s top-of-mind with C-suite executives.

Secondly, they state that through 2021, just 2 years from now, 65% of Infrastructure and Operations leaders will underinvest in their availability and recovery needs because they use estimated cost-of-downtime metrics.

As it turns out, Ayehu can help you get a more accurate estimate of your downtime costs so they’re not underestimated.

In our eBook titled “How to Measure IT Process Automation ROI”, there’s a specific formula for calculating the cost of downtime. The eBook is free to download on our website, and also includes access to all of our ROI formulas, which are fairly straightforward to calculate.

Let’s look at another data point about outages, this one from the Uptime Institute’s 2019 Annual Data Center Survey Results. They report that “Outages continue to cause significant problems for operators. Just over a third (34%) of all respondents had an outage or severe IT service degradation in the past year, while half (50%) had an outage or severe IT service degradation in the past three years.”

So if you were thinking painful outages only happen at your organization, think again. They’re happening everywhere. And as the research from Statista emphasized, when outages hit, it’s usually very expensive.

The Uptime Institute has an even more alarming statistic they’ve published.

They’ve found that more than 70% of all data center outages are caused by human error and not by a fault in the infrastructure design!

Let’s pause for a moment to ponder that. In 70% of cases, all it took to bring today’s most powerful high-tech to its knees was a person making an honest mistake.

That’s actually not too surprising though, is it? All of us have mistyped a keyboard stroke here or made an erroneous mouse click there. How many times has it happened that someone absent-mindedly pressed “Reply All” to an email meant for one person, then realized with horror that their message just went out to the entire organization?

So mistakes happen to everyone, and that includes data center operators. And unfortunately, when they make a mistake that leads to an outage, the consequences can be catastrophic.

One well-known example of an honest human mistake that led to a spectacular outage occurred back in late February of 2017. Someone on Amazon’s S3 team input a command incorrectly that led to the entire Amazon Simple Storage Service being taken down, which impacted 150,000 organizations and led to many millions of dollars in losses.

If infrastructure design usually isn’t the issue, and 70% of the time outages are a direct result of human error, then logic suggests that the key would be to eliminate the potential for human error. And just to emphasize the nuance of this point, we’re NOT advocating eliminating humans, but eliminating the potential for human error while keeping humans very much involved. How do we do that?

Well, you won’t be too surprised to learn we do it through automation.

Let’s start by taking a look at the typical infrastructure and operations troubleshooting process.

This process should look pretty familiar to you.

In general, many organizations (including large ones) do most of these phases manually. The problem with that is that it makes every phase of this process vulnerable to human error.

There’s a better way, however. It involves automating much of this process, which can reduce the time it takes to remediate an IT incident down to seconds. And automation isn’t just faster, it also eliminates the potential for human error, which should radically reduce the likelihood that your environment will experience an outage due to human error.

Here’s how that would work. It involves using the Ayehu platform as an integration hub in your environment. Ayehu would then connect to every system that needs to be interacted with when remediating an incident.

For example, if your environment has a monitoring system like SolarWinds, Big Panda, or Microsoft System Center, that’s where an incident will be detected first. The monitoring system (now integrated with Ayehu) will generate an alert which Ayehu will instantaneously intercept. (BTW – if there’s a monitoring system or any kind of platform in your environment that we don’t have an off-the-shelf integration for, it’s usually still pretty easy to connect to it via a REST API call.)

Ayehu will then parse that alert to determine what the underlying incident is, and launch an automated workflow to remediate it.

As a first step in our workflow we’re going to automatically create a ticket in ServiceNow, BMC Remedy, JIRA, or any ITSM platform you prefer. Here again is where automation really shines over taking the manual approach, because letting the workflow handle the documentation will ensure that it gets done in a timely manner (in fact, in real-time) and that it gets done thoroughly. This brings relief to service desk staff who often don’t have the time or the patience to document every aspect of a resolution properly because they’re under such a heavy workload.

The next step, and actually this can be at any step within that workflow, is pausing its execution to notify and seek human approval for continuation. To illustrate why you might do this, let’s say that a workflow got triggered because SolarWinds generated an alert that a server dropped below 10% free disk space. The workflow could then go and delete a bunch of temp files, it could compress a bunch of log files and move them somewhere else, and do all sorts of other things to free up space. Before it does any of that though, the workflow can be configured to require human approval for any of those steps.

The human can either grant or deny approval so the workflow can continue on, and that decision can be delivered via laptop, smartphone, email, instant messenger, or even regular telephone. However, please note that this notification/approval phase is entirely optional. You can also choose to put the workflow on autopilot and proceed without any human intervention. It’s all up to you, and either option is easy to implement.

Then the workflow can begin remediating the incident which triggered the alert.

As the remediation is taking place, Ayehu can update the service desk ticket in real-time by documenting every step of the incident remediation process.

Once the incident remediation is completed, Ayehu can automatically close the ticket.

Finally, Ayehu can go back into the monitoring system and automatically dismiss the alert that triggered the entire process.

This, by the way, illustrates why we think of Ayehu as a virtual operator which we sometimes refer to as “Level 0 Tech Support”. A lot of incidents can be resolved automatically by Ayehu without any human intervention, and thus without the need for attention from a Level 1 technician.

This then is how you can go from alert to remediation in 15 seconds, while simultaneously eliminating the potential for human error that can lead to outages in your environment.

Gartner concurs with this approach.

In a recently refreshed paper they published (ID G00336149 – April 11, 2019) one of their Vice-Presidents wrote that “The intricacy of access layer network decisions and the aggravation of end-user downtime are more than IT organizations can handle. Infrastructure and operations leaders must implement automation and artificial intelligence solutions to reduce mundane tasks and lost productivity.”

No ambiguity there.

Gartner’s advice is a good opportunity for me to segue into one last topic – artificial intelligence.

The Ayehu platform has AI built-in, and it’s one of the reasons you’ll be able to not only quickly remediate your IT incidents, but also quickly build the workflows that will do that remediation.

Ayehu is partnered with SRI International (SRI), formerly known as the Stanford Research Institute. In case you’re not familiar with them, SRI does high-level research for government agencies, commercial organizations, and private foundations. They also license their technologies, form strategic partnerships (like the one they have with us) and creates spin-off companies. They’ve received more than 4,000 patents and patent applications worldwide to date. SRI is our design partner, and they’ve designed the algorithms and other elements of our AI/ML functionality. What they’ve done so far is pretty cool, but what we’re working on going forward is what’s really exciting.

One of the ways Ayehu implements AI is through VSAs, which is shorthand for “Virtual Support Agents”.

VSA’s differ from chatbots in that they’re not only conversational, but more importantly they’re also actionable. That makes them the next logical step or evolution up from a chatbot. However, in order for a VSA to execute actionable tasks and be functionally useful, it has to be plugged in to an enterprise grade automation platform that can carry out a user’s request intelligently.

We deliver a lot of our VSA functionality through Slack, and we also have integrations with Alexa and IBM Watson. We’re also incorporating an MS-Teams interface, and looking into others as well.

How is this relevant to remediating incidents?

Well, if a service desk can offload a larger portion of its tickets to VSA’s, and provide its users with more of a self-service modality, then that frees up the service desk staff to automate more of the kinds of data center tasks that are tedious, repetitive, and prone to human error. And as I’ve previously stated, eliminating the potential for human error is key to reducing the likelihood of outages.

Speaking of tickets, another informal webinar poll we conducted asked:

On average, how many support tickets per month does your IT organization deal with?

  • Less than 100
  • 101 – 250
  • 251 – 1,000
  • More than 1,000

Here’s how our audience responded:

Nearly 90% receive 251 or more tickets per month. Over half get more than 1,000!

For comparison, the Zendesk Benchmark reports that among their customers, the average is 777 tickets per month.

Given the volume of tickets received per month, the current average duration it takes to remediate an incident, and most importantly the onerous cost of downtime, automation can go a long way towards helping service desks maximize their efficiency by being a force multiplier for existing staff.

Q:          What types of notifications can the VSA send at the time of incident?

A:           Notifications can be delivered either as text or speech.

Q:          How does the Ayehu tool differ from other leading RPA tools available on the market?

A:           RPA tools are typically doing screen automation with an agent. Ayehu’s automation is an agentless platform that primarily interfaces with backend APIs.

Q:          Do we have to do API programming or other scripting as a part of implementation?

A:           No. Ayehu’s out-of-the-box integrations typically only require a few configuration parameters.

Q:          Do we have an option to create custom activities? If so, which programing language should be used?

A:           In our roadmap, we will be offering the ability to create custom activity content out-of-the-box.

Q:          Do out-of-the-box workflows work on all types of operating systems?

A:           Yes. You just define the type of operating system within the workflow.

Q:          How does Ayehu connect and authenticate with various endpoint devices (e.g. Windows, UNIX, network devices, etc.)? Is it password-less, connection through a password vault, etc?

A:           That depends on what type of authentication is required internally by the organization. Ayehu integrated with the CyberArk password vault can be leveraged when privileged account credentials are involved. Any type of user credential information that is manually input into a workflow or device is encrypted within Ayehu’s database. Also, certificates on SSH commands, Windows authentication, and localized authentication are all accessible out-of-the-box. Please contact us for questions about security scenarios specific to your environment.

Q:          What are all the possible modes that VSAs can interact with End Users?

A:           Text, Text-to-Speech, and Buttons.

Q:          Can we create role-based access for Ayehu?

A:           Yes. That’s a standard function which can also be controlled by and synchronized with Active Directory groups out-of-the-box.

Q:          Apart from incident tickets, does Aheyu operate on request tickets (e.g. on-demand access management, software requests from end-users, etc.)?

A:           Yes. The integration packs we offer for ServiceNow, JIRA, BMC Remedy, etc. all provide this capability for both standard and custom forms.

Q:          Does Ayehu provide APIs for an integration that’s not available out of the box?

A:           Yes. There are two options. You can either forward an event to Ayehu using our webservice which is based on a RESTful API, or from within the workflow you can send messages outbound that are either scheduled or event-driven. This allows you to do things such as make a database call, set an SNMP trap, handling SYSLOG messages, etc.

Q:          Does Ayehu provide any learning portal for developers to learn how to use the tool?

A:           Yes. The Ayehu Automation Academy is an online Learning Management System we just launched recently. It includes exams that provide you an opportunity to bolster your professional credentials by earning a certification. If you’re looking to advance your organization’s move to an automated future, as well as your career prospects, be sure to check out the Academy.

Q:          Does Ayehu identify issues like a monitoring tool does?

A:           Ayehu is not a monitoring tool like Solarwinds, Big Panda, etc. Once Ayehu receives an alert from one of those monitoring systems, it can trigger a workflow that remediates the underlying incident which generated that alert.

Q:          We have 7 different monitoring systems in our environment. Can Ayehu accept alerts from all of them simultaneously?

A:           Yes. Ayehu’s integrations are independent of one another, and it can also accept alerts from webservices. We have numerous deployments where thousands of alerts are received from a variety of sources and Ayehu can scale up to handle them all.

Q:          What does the AI in Ayehu do?

A:           There are different areas where AI is used. From use in understanding intent through chatbots to workflow design recommendations, and also suggesting workflows to remediate events through the Ayehu Brain service. Please contact an account executive to learn more.

New call-to-action

The Future of MSPs: The Good, The Bad, The Ugly

/in , , , /by

Author: Guy Nadivi, Sr. Marketing Director, Ayehu

Leading analyst firms are forecasting a lot of turmoil ahead for MSPs, so I thought it would be well worth exploring not only the ramifications of this expected market upheaval, but also how smart MSPs can actually take advantage of it.

Are any of you fans of old westerns?  I’m personally a big fan of movies about the old west, an interesting time in American history.  Western cinema, as many of you already know, often depicts harsh wilderness landscapes where people end up in a shootout, fighting over something that’s important to them.

One of the best examples of Western cinema is a classic movie called “The Good, The Bad, & The Ugly”.  Maybe some of you have seen it.  Back when I still purchased DVDs, this was actually the first DVD I ever bought. 

This movie title, “The Good, The Bad, & The Ugly” is an apt metaphor I’m going to use to help me describe the current situation in the MSP market, where much like the old west, many MSPs find themselves in a shootout over something very important – market share.

Let’s start with ‘The Good.’  According to MarketsAndMarkets, a research and advisory firm focused on B2B markets, “The managed services market is expected to grow from over $180 billion in 2018 to $282.0 billion by 2023, a Compound Annual Growth Rate of 9.3%.”  That’s a pretty good market to be in.

Here’s the ‘The Bad,’ though. According to Gartner, “…as competition heightens, service providers will be forced to aggressively roll automation out across their client base and service lines because, if an existing provider is slow in implementing automation, this will leave that account quite vulnerable to competition, proposing a strong artificial intelligence proposition with the possibility of a vendor switch.”  In other words, Gartner’s saying that MSPs who don’t start introducing automation & AI to their customers, are now at risk of being left by that customer for another MSP that does.

And here’s ‘The Ugly.’ Again this is from Gartner and please note, this is advice they’re giving to sourcing executives at enterprises that hire MSPs.  “Understand the provider’s service capabilities, product development plans and AI roadmap to be able to negotiate effective reductions associated with new technology. Otherwise, consider moving away from the service provider if investments are lacking, lagging, or the service provider is not actively sharing the benefits with the client.” 

That last part might require a bit of explanation. Gartner is telling customers that they should EXPECT their MSP vendor to start automating their service offerings, and begin sharing the benefits of automation back to the client in the form of reduced charges. If an MSP doesn’t do that, they’re explicitly telling sourcing executives to go find another MSP that does!

Some of you might now be realizing seeing why I’m using the old west as a metaphor for the MSP market.  There’s a big shootout coming among MSPs that don’t start rolling out effective automation for their clients.

Here’s another one. The Good – according to Gartner, “Many of the large players in the Gartner Magic Quadrants that address IT infrastructure have rolled out intelligent automation that provides for effective management of the data center, end user, service desk and applications. The rollouts have been in the operations area and have reported improvements of 30% cost savings with 30% gain in service quality.”  So the organizations that have started automating are seeing significant benefits, meaning that automation is working really well.

But, also according to Gartner, here’s the The Bad – “Reconcile with the fact that revenue cannibalization is bound to happen in the near term because of automation. Instead, prepare to divert cost savings into fueling new projects. This is the best way to protect your turf.”  Interpretation: Gartner is telling MSPs that automation will cause unavoidable revenue losses in the immediate future, but your best bet for safeguarding market share is to invest in new automation projects now.

Finally, here’s The Ugly, and this too is from Gartner, “Use forward pricing to reap the benefits of artificial intelligence in your infrastructure outsourcing deal.” BTW – this is another Gartner recommendation specifically addressed to sourcing executives.  They’re advising them to incorporate the expected savings of artificial intelligence, and by inference automation, into their outsourcing deals REGARDLESS of whether or not their infrastructure provider offers it.  Meaning that whether or not automation and AI are part of an MSP’s strategy, the market will be expecting it to be, and that will put further downward pricing pressure on a business with already thinning margins.

Let’s do just one more of these.

The Good.  Gartner says, “Those providers that invest early will see their business flourish for a few years, and then will land in a position where the business around those services is underpinned by a positive and sustainable margin performance. Yes, it will be transactional and maybe per quantity in nature, but it will be nevertheless sustainable.”  So if you make the investment in automation now, you will reap the profitable benefits down the road in a SUSTAINABLE way.

Here’s The Bad.  “Those that fail to invest will see quick revenue erosion, followed by margin erosion, because they will be forced to lower prices without being able to enjoy the reduction in delivery costs that automation can offer.”  So again, they’re saying the market is expecting MSPs to provide automation, and will also expect lower pricing going forward, regardless of whether or not the MSP even offers automation.

And here’s the last Ugly.  “It will not be a question of getting to ‘smaller but sustainable,’ but a case of exiting with a strong focus on damage limitation.”  This advice from Gartner basically boils down to a warning that if you’re not going to start using and offering automation soon, you should consider getting out now while you can still get some value for your business.  Pretty dire warning!

Now that you’ve heard what the experts think, if you’re an MSP is it time to hit the panic button? 

NO!  Don’t panic.  Not yet anyways.

Let’s return again to our old west theme.  Back in those days when people went to the local saloon to enjoy some recreational fun, everybody played the same card game just like they always did. 

In modern times, up until recently, the game was always the same for MSPs too.  Now though, the MSP game is changing.

In fact, thanks to automation and other technologies, the MSP game is changing dramatically and everybody is being dealt a new hand. If you’re an MSP, your new hand in this new game means a new opportunity to increase market share!

Just to be absolutely clear about the changing game for MSPs, let me illustrate what exactly is changing.

Traditionally, the MSP game was about filling up massive cubicle farms to provide services with inexpensive labor.  Unfortunately, that’s just not sustainable anymore. Even if you’ve got a supply of the absolute cheapest labor and you can double up people in each cubicle, there’s one inconvenient fact that can’t be escaped – people don’t scale very well.

That includes even your very best data center workers, who can only handle so much. Today, analysts and thought leaders are telling companies to walk away from these kinds of outsourced deals, no matter how cheap your labor is.

And why are they recommending that?

Because the new reality is that digital labor is MUCH cheaper.  Not only that, but digital labor takes no vacations, or coffee breaks, or sick time, and it never has mood swings.  It’s always available, 24/7/365 and unlike people, it is extremely scalable.

The new game for MSPs is Automation-as-a-Service.  Leveraging digital labor to provide a much better offering, and doing it for even less than before.

Remember, in this new game MSPs are expected to play, Gartner and others are telling your customers that what they should demand from you is more quality, increased speed, and better results, all at a lower cost.  The only practical way an MSP can do that is with automation.

Back once more to the old west.  One of the really great stories in the history of that time period was the gold rush.  The gold rush of the 1800’s made a lot of MSPs very rich.  Yes, you read that right – MSPs.  Except back then, M.S.P. stood for Mineral Searching Prospectors.  (Alright, maybe I’m the only one who thought that was kind of funny.)

Today’s gold rush doesn’t involve any shovels or pick axes or specialized pans for sifting gold nuggets out of rivers.  That’s because today’s gold rush is in automation powered by AI.  Ayehu predicts that between now and about the middle of this century, a lot of MSP’s are going to get very rich by using an enterprise automation platform to provide Automation-as-a-Service for their customers.

Before diving into that though, I’d like to talk just a little bit about open source automation.

If there’s one character from the old west that best epitomizes the idea of working with open source software, it’s undoubtedly the blacksmith.  Everybody knows what blacksmiths did back then, right?  They took a piece of metal, and forged it into something like a horseshoe.  And by forging I mean they did a lot of hammering and a lot of sweating.

That’s exactly what you’re going to do with open source software.  Except instead of hammering, you’re going to be doing a lot of coding, but you’ll still probably do a lot of sweating too, just like the blacksmith. 

Maybe that’s appealing on some level.  Build it yourself and take full pride in forging an automation tool that does exactly what you want.  Except the problem there is that while you’re hammering away on your keyboard just to build the automation tool itself, your competitors are using commercial-grade automation software like Ayehu that works right out-of-the-box and is fully supported by the publisher.  That means your competitors are orchestrating actual workflows for their customers that are up & running quickly and in production to start earning those customers an ROI.  The best way to stay competitive as an MSP is to go with the tool that’s already proven itself and can earn a fast ROI for your customers.

Remember – generally speaking, your customers aren’t worrying about the plumbing.  They just want you to give them hot water.

Let’s discuss a couple of use case examples.

The first use case is a major international financial services firm, with what can only be described as a colossal environment:

  • They have over 60,000 servers
  • They also have over 10,000 database instances
  • And they have nearly 500 supported applications!

Their challenge was the high cost of monitoring and maintaining this massive infrastructure.

Using Ayehu’s automation platform, they realized:

  • A 40% improvement in MTTR
  • A 90% improvement in response times
  • And together, those two yielded a 15% cost savings in year one!

Not a bad return on investment, and a huge win for our MSP partner that delivered these results to this customer.

The second use case involves one of the largest department stores in America.  Not quite as big as the previous company, but pretty big nonetheless.  Their infrastructure included:

  • About 20,000 servers
  • Nearly 6,500 database instances
  • And all this was spread out between 2 different datacenters!

Their staff was spending a lot of time and effort on manual, repetitive tasks that were impacting their resolution times.

After Ayehu was deployed, they experienced:

  • a 95% improvement in MTTR
  • a 1,500 man-hour reduction of effort in Year 1
  • and a savings of nearly half-a-million dollars!

Quite an impact.

The final case study I want to share with you shows the power of automation in reducing the cost of operations for the MSP.  This case study comes from a global MSP partner of ours who’s among the largest $ multi-billion MSPs.

They were looking to reduce operations costs and improve their margins at one particular client where they had a multi-year contract with a project value of $11.6 Million per year.

After implementing Ayehu at that customer to automate numerous repetitive manual processes, their operational costs steadily dropped each year until by the 3rd year of their engagement, they were saving 35% in costs using Ayehu automation, all of which dropped straight to their bottom line.

And thanks to Ayehu, they were able to deliver a 30% FTE optimization while increasing their SLA performance by 98%.

As you can imagine, now that this MSP has mastered our automation platform with such success, they’re going to be aggressively competitive in the market place.

Speaking of SLAs, I should also point out that incorporating automation into your managed service practice will allow you to say goodbye to SLA penalties and missed targets. As previously mentioned, automation never takes a break, and it also remediates incidents much faster.  That more than anything will give your MSP practice its best shot at hitting its KPI goals. Typically with automation, you can reduce ticket-handling time for incidents down to seconds.

BTW – Since offering automation will alter your cost structure as it did for the MSP above, it will open up many more opportunities for you that were not previously economically profitable.  Automation will also enable you to generate more business opportunities from your existing customer base.  On average, our partners tell us Ayehu has increased their MSP wins by about 10x.

Q:          What’s your onboarding program like & how long does it take?

A:           Onboarding generally takes 6 weeks.  During that time we’ll put your team through training, help you get your own Ayehu environment up & running, and hold your hands helping you build your first workflows.  We’ll also help you build POC’s with your clients, and enable your success however we can.

Q:          What’s the difference between your solution and a freebie Open Source Software download?

A:           It depends on what open source software you’re referring to.  In general though, open source software means you’re doing all the heavy lifting of building out your own tool.  So be prepared to do a LOT of coding.  We’ve invested over a decade of man hours building out the Ayehu automation platform and it’s ready to go out-of-the-box right now without any coding.  The first question you should ask yourself then is, would you rather invest your time & effort reinventing the wheel, or using the wheel that’s been on the market for over 10 years to start adding value to your clients from day one?

Q:          How should an MSP determine when to use Ayehu versus some other automation tool?

A:           That depends on what it is you want to automate.  There’s a lot of different automation tools out there with a lot of different specialties.  Ayehu has a very specific focus on automating IT & Security operations.  We’ve been doing it a long time, we’re very good at it, and we’d be a great choice for any MSP looking for that kind of solution.

Q:          What is the minimum time to learn Ayehu?

A:           Very minimal.  Usually hours, but no more than a couple days.  We like to tell people all the time – take your lowest-level SysAdmin (even an intern), preferably someone who’s never written a single line of code in their lives, and let us train them for just one day.  Afterwards, they’ll probably end up being the most productive person on your IT staff.  Ayehu is very easy to learn.  If you’ve ever used a tool like Visio to build something like an org chart, then you’re already pretty well qualified to build automated workflows with Ayehu.

Q:          What makes Ayehu a platform that MSPs should use, compared to other automation tools?

A:           The Ayehu automation platform is actually designed with MSPs in mind.  So that means features like:

  • Being SaaS-ready which allows an MSP to create their own automation cloud, and since it’s also multi-tenant that means you can partition the same automation cloud out to different customers while managing it all centrally from one instance.  We deliberately made Ayehu an enabler for MSPs that makes it easy for them to offer Automation-as-a-Service.
  • Providing white labeling, so you can rebrand Ayehu as your own tool, which is a great way to reinforce brand loyalty with your customers.

•            Offering a strong partner enablement service that gets you up & running quickly so you can start delivering value to your clients ASAP & begin conquering more market share with automation.

Q:          You mentioned that Ayehu includes AI, but you didn’t give much detail.  Can you please elaborate on what Ayehu’s AI capabilities are?

A:           Ayehu is partnered with SRI International, formerly known as the Stanford Research Institute.  SRI holds something like 4,000 patents worldwide including for things like the original mouse & SIRI, Apple’s conversational AI.  SRI is Ayehu’s design partner, and they’ve designed a lot of the really cool stuff like Machine Learning-driven Dynamic Activity Suggestions.  That means that based on the workflow you’re building, our system provides a real-time recommendation on the next best activity to incorporate into your workflow, based on what we know has worked best for other customers building similar workflows.  That’s been available since last year.

              Another cool AI/ML feature is Dynamic Rule Suggestions, to augment the current static rules we have for triggering workflows. What that means is that when an incident comes into Ayehu, if we have a static rule that matches its profile, then that rule will kick off a workflow to remediate that incident.  Dynamic Rule Suggestions will allow us to suggest rules for incidents that don’t match any rules so they don’t just fall through the cracks.

              By next year, we’ll be offering Dynamic Workflow Suggestions. This is exactly what it sounds like, namely we’re going to be offering real-time suggestions of best practice workflows and workflows that are industry-specific, based on a little understanding of what you’re trying to accomplish.

              So there’s a lot of very cool AI & machine learning features baked into the product and we believe it’s all going to give our customers an insurmountable market advantage.

To see this information in action, click the image below to watch the on-demand webinar.

New call-to-action

How to Automate Incident Response for Splunk Alerts in Minutes

/in , , /by

Author: Guy Nadivi, Sr. Director of Marketing, Ayehu

Let’s talk about Splunk, a market leader in the Security Event Information Management (SEIM) market. BTW – You can always tell who the market leader is in any category when its competition starts touting itself as the ones who will eradicate that company. Recently, one of Splunk’s competitors described themselves as “Splunk Killers”, reaffirming that Splunk is indeed at the head of its class in that segment.

In Gartner’s 2018 Magic Quadrant for the SIEM market, Splunk appears higher than everyone else and further to the right than anybody but IBM. What this means is they excel above all other competitors on the y-axis of the Magic Quadrant, which is a measurement of “Ability to Execute.” On the x-axis of the Magic Quadrant, which measures “Completeness of Vision,” they exceed almost everyone except IBM.

Score highly on those two measurements, and Gartner considers you a Market Leader.

Market share is another key indicator of market leadership, and here Splunk is ranked No. 2 with 13.7% market share. Only IBM has a larger market share when it comes to SEIM’s.

Thanks to Splunk’s January 31, 2019 Form 10-K filing with the SEC, we also know they have 17,500 customers in more than 130 countries, including 90% of the Fortune 100. Another clear indication that they are a leader in this market.

With a market position like that, it seems worthwhile to talk about how to quickly and easily automate incident remediation for Splunk alerts in minutes.

As many people know, Splunk produces software for capturing, indexing, correlating, searching, monitoring, and analyzing machine-generated big data.

Some sources of that data include logs for Windows events, Web servers, and live applications, as well as network feeds, metrics, change monitoring, message queues, archive files, and so on.

Generally, these data sources can be categorized as:

  • Files and directories
  • Network events
  • Windows sources
  • And the catch-all category of “other sources”

There are a number of outputs and outcomes Splunk generates from this data, including:

  • Analyzing system performance
  • Troubleshooting failure conditions
  • Monitoring business metrics
  • Creating dashboards to visualize and analyze results
  • And of course storing and retrieving data for later use

That’s A LOT of data, and the more systems Splunk monitors, and the more those systems grow, the greater the volume of machine data that gets generated. This is becoming a problem because IT and security operations are getting inundated by all this data, and not just from Splunk, but other systems as well, though Splunk generates a big chunk of this.

Every time there’s an incident, an event, a threshold being crossed, etc. new data is generated, adding to the surge already flooding over IT and Security Operations. And it’s only getting worse.

Ultimately, it’s people who have to deal with all this data, and the problem is, (as we often say) people don’t scale very well.

Even the very best data center workers in NOCs and SOCs can only handle so much. At some point – and that point is pretty much right now – automation has to take on a greater share of the task burden all this growth in data is necessitating.

Why automation? Because people may not scale very well, but automation DOES! And if you’re in one of these overwhelmed data centers, that should be music to your ears.

Here are just a few of the ways automation can bring relief to NOCs and SOCs drowning in Splunk data:

Triggering Workflows

Let’s say there’s been an event detected of a corporate website being hacked and defaced. This event can trigger an automatic workflow that quickly restores a website to its pre-defacement state. In fact, an automation platform like Ayehu can do this MUCH quicker than humans could do manually once they got the alert. Restoring the website automatically and almost instantaneously minimizes the damage to corporate reputation, not to mention the threat to job security because the defacement happened in the first place.

Remediating Incidents

In addition to the example of remediating a website defacement incident, let’s consider a situation where Splunk generates an alert about a specific machine due to some observed suspicious activity. Ayehu can remotely lock it either automatically or at the SOC analyst’s manual command, to mitigate any damage until a hands-on inspection can take place. Furthermore, this automated incident remediation workflow could also include doing things like deactivating that user’s Active Directory credentials, turning off their card key’s ability to swipe in or out of a building, etc.

Data Enrichment

This task is well known to anyone who’s ever had to perform cybersecurity forensics during and after an incident. It involves aggregating all the information a SOC analyst needs to make an informed decision about what’s happening in real-time, or what happened as part of an after-incident evaluation. This can be a laborious manual task, and certainly one that’s difficult to script out.

If your automation platform easily integrates with just about anything in a typical, heterogeneous IT environment, however, then it can gather this critical information very rapidly as well as add more precise context to it about the nature of the incident. This will greatly reduce time-to-decision-making for SOC analysts, which is vital when, for example, you’re watching a ransomware virus swiftly encrypt your enterprise data and you need to decide on a course of action fast.

Opening Tickets

Just about every data center uses an ITSM platform like ServiceNow, JIRA, BMC Remedy, or one of many others. It’s very important to document what steps were taken to remediate an incident or conduct a cybersecurity forensics investigation. SOC analysts are pretty overwhelmed these days, and often don’t have the time to do that. When they do have time, they often don’t document as thoroughly as necessary in order to provide a complete picture of what transpired.

An automation tool like Ayehu can do this much quicker, and in real-time during workflow execution, so everything is properly documented, and nothing slips through the cracks.

Now let’s walk through the flow of events that uses Splunk data and alerts as triggers for actions.

We call this flow a closed-loop, automated incident management process. It starts out with Ayehu NG creating an integration between Splunk and whatever IT Service Management or help desk platform you’re using, be it ServiceNow, JIRA, BMC Remedy, etc.

When Splunk generates an alert or any kind of data you want to act upon, Ayehu intercepts it via the integration point. It will then parse it to determine the underlying incident, and launch the appropriate workflow for that situation, whether it be remediating that specific underlying incident, gathering information for forensic analysis, or whatever.

While this is taking place, Ayehu also automatically creates a ticket in your ITSM, and updates it in real-time by documenting every step of the workflow. Once the workflow is done executing, Ayehu automatically closes the ticket. All of this can occur without any human intervention, or you can choose to keep humans in the loop.

This closed-loop illustration also reveals why we think of Ayehu as a virtual operator, which we sometimes refer to as “Level 0 Tech Support”. Many incidents can simply be resolved automatically by Ayehu without human intervention, and without the need for attention from a Level 1 technician.

Imagine automating manual processes like Capture, Triage, Enrich, Respond, and Communicate. Automating resolution and remediation can result in a pretty significant savings of time, which can be particularly critical for data centers feeling overwhelmed.

Customers tell us over and over that automating the manual, tedious, time-intensive stuff accelerated their incident resolution by 90% or more.

We can also say with confidence that you can automate incident response for Splunk alerts in minutes, because Ayehu’s automation platform is agentless. Being agentless also makes us non-intrusive since we leverage API’s, SSH, and HTTPS behind enterprise firewalls under that organization’s security policy to perform automation. The only software to install is on a server, either physical or virtual, which centralizes management and greatly simplifies maintenance and upgrades.

Another reason it only takes minutes to automate incident response for Splunk alerts is because the Ayehu automation platform is codeless. This is something really important to consider because while there are many vendors out there touting their platforms as “automation”, the fact remains that they’re really just frameworks for scripting, and we steadfastly believe that scripting IS NOT automation.

For starters, in order to script you need to have programming expertise. With a true automation tool, however, you shouldn’t need to have any programming expertise. In fact, the automation platform should be so easy to use, even a junior SysAdmin with zero programming expertise should be able to master it in less than a day. Why is that so important? Because one of the promises of true automation is that you don’t have to rely on specialized talent to orchestrate activities in your environment. Requiring specialized programmers would be a bottle-neck to that goal.

Finally, the Ayehu automation platform includes AI and Machine Learning built into the product.

The first thing you should know about Ayehu’s AI and Machine Learning efforts is that we’re partnered with SRI International (SRI), formerly known as the Stanford Research Institute. For those not familiar, SRI does high-level research for government agencies, commercial organizations, and private foundations. They also license their technologies, form strategic partnerships (like the one they have with us), and create spin-off companies. They’ve received more than 4,000 patents and patent applications worldwide to date. SRI is our design partner, and they’ve designed the algorithms and other elements of our AI/ML functionality. What they’ve done so far is pretty cool, but what we’re working on going forward is really exciting.

Questions and Answers

Q:          What are the pros and cons of using general purpose bot engines compared to your solution?

A:           General purpose bot engines won’t actually perform the actions on your infrastructure, devices, monitoring tools, business applications, etc. All they can really do is ingest a request. By contrast, Ayehu not only ingests requests, but actually executes the necessary actions needed to fulfill those requests. This adds a virtual operator to your environment that’s available 24x7x365. Additionally, Ayehu is a vendor-agnostic tool that interfaces with MS-Teams, Skype, etc. to provide these general purpose chat tools with intelligent automation capabilities.

Q:          Do you have an on-premise solution?

A:           Yes. Ayehu can be installed on-premise, on a public or private cloud, or in a hybrid combination of all three.

Q:          Do you have voice integration?

A:           Ayehu integrates with Amazon Alexa, and now also offers Angie™, a voice-enabled Intelligent Virtual Support Agent for IT Service Desks.

Q:          If a user selects a wrong choice (clicks the wrong button) how does he or she fix it?

A:           It depends on how the workflow is designed. Breakpoints can be inserted in the workflow to ask the endpoint user to confirm their button selection, or go back to reselect. Ayehu also offers error-handling mechanisms within the workflow itself.

Q:          Does Ayehu provide orchestration capabilities or do you rely on a 3rd party orchestration tool?

A:           Ayehu IS an enterprise-grade orchestration tool, offering over 500 pre-built platform-specific activities that allow you to orchestrate multi-platform workflows from a single pane of glass.

Q:          Can you explain in a bit more detail on intent-based interactions?

A:           Intent is just that, what the user’s intent is when interacting with the Virtual Support Agent (VSA). For example, if a user types “Change my password”, the intent could be categorized as “Password Reset”. That would then trigger the “Password Reset” workflow.

Q:          Thanks for the information so far, great content! I would like to know if I can use machine learning from an external source, train my model, and let Ayehu query my external source for additional information?

A:           Yes. Ayehu can integrate with any external source or application, especially when it has an API for us to interface with.

Q:          Can I create new automations to my inhouse applications?

A:           Yes. Ayehu can integrate with any application bi-directionally. Once integrated with your inhouse applications, Ayehu can execute automated actions upon them.

Q:          Is there an auto form-filling feature? (which can fill in a form in an existing web application)

A:           Yes. Ayehu provides a self-service capability that will allow this.

Q:          How can I improve or check how my workflows are working and helping my employees to resolve their issues?

A:           Ayehu provides an audit trail and reporting that provides visibility into workflow performance. Additionally, reports are available on time saved, ROI, MTTR, etc. that can quantify the benefits of those workflows.

Q:          What happens when your VSA cannot help the end user?

A:           The workflow behind the VSA can be configured to escalate to a live support agent.

Q:          If there is a long list of choices – what options do you have? Dropdown?

A:           In addition to the buttons, dropdowns will be provided soon in Slack as well.

Q:          Did I understand correctly, an admin will need to create the questions and button responses? If so, is this a scripted Virtual Agent to manage routine questions?

A:           Ayehu is scriptless and codeless. The workflow behind the VSA is configured to mimic the actions of a live support agent, which requires you to pre-configure the questions and expected answers in a deterministic manner.

Q:          Is NLP/NLU dependent on IBM Watson to understand intent?

A:           Yes, and soon Ayehu will be providing its own NLP/NLU services.

Q:          Are you using machine learning for creating the conversations? Or do I have to use intents and entities along with the dialogs?

A:           Yes, you currently have to use intents and entities, but our road map includes using machine learning to provide suggestions that will improve the dialogs.

Q:          What are the other platforms that I can deploy the VSA apart from Slack?

A:           Microsoft Teams, Amazon Alexa, ServiceNow ConnectNow, LogMeIn, and any other chatbot using APIs.

This is a recap of a live Webinar we hosted in May 2019. To watch the on-demand recording and see this content in action, please click here.

New call-to-action

Creating an Intelligent Virtual Support Agent for your ITSM

/in , , , /by

Author: Guy Nadivi, Sr. Director of Marketing, Ayehu

A topic that’s really getting a lot of buzz these days is Virtual Support Agents.  Virtual Support Agents – or VSAs – are the next logical evolution of a chatbot, because where chatbots are primarily conversational, a VSA is both conversational and actionable, making them much more valuable to an enterprise, and particularly to a service desk using an ITSM platform.

In order to better understand why VSAs are so top-of-mind, it can be helpful to take a step back and understand what’s happening right now in IT operations at enterprises around the world, particularly with all the data they’re dealing with.

Do you know what a Zettabyte is? NO GOOGLING!

A Zettabyte is a trillion Gigabytes. That’s a “1” followed by 21 zeroes.

As humans, it can be hard for us to wrap our minds around numbers that large, so let’s use a visual metaphor to help provide a frame of reference for how much data 1 Zettabyte represents.

The visual metaphor we’d like you to envision should be a familiar one – grains of rice. For the sake of this visual comparison, let’s say one grain of rice is equal to one byte of data.

That makes a kilobyte, a thousand grains, equal to about a CUP of rice.

A megabyte of data would then be the equivalent of 8 BAGS of rice.

1 gigabyte of data in terms of rice is equal to 3 container TRUCKS.

A terabyte of data would then be the equivalent of 2 CONTAINER SHIPS full of nothing but rice.

Now get this – an exabyte of data in terms of rice, would cover all of MANHATTAN.

A petabyte of data, would just about cover all of TURKEY. (BTW – turkey & rice is a great combo!)

Finally, here’s what a Zettabyte of data in terms of rice would do.  It would fill up the PACIFIC OCEAN! 

Now, this last visual using the Pacific Ocean is very relevant, especially if you work in IT operations.  That’s because you literally feel like an entire ocean’s worth of data is inundating you these days, thanks to all the systems you’re maintaining that create, store, access and deliver data for your employees, customers, partners, etc.  Life in IT operations is a relentless tsunami of incidents, events, thresholds being crossed, etc., and it’s only getting worse.

How much worse?

In 2017 The Economist published a chart produced by IDC in conjunction with Bloomberg estimating the size of data comprising “The Digital Universe”. In 2013 there were 4.4 Zettabytes of data worldwide, but by 2020 there will be 44 Zettabytes. That’s an astounding CAGR of 47%. Don’t expect things to slow down though, because it’s estimated that by 2025 there will be 175 zettabytes of data worldwide!

Interestingly, IDC/Bloomberg discovered what appears to be a correlation between the exponential growth of data and an increase in the number of times companies are mentioning “artificial intelligence” in their earnings calls. This is probably not a coincidence, and it underscores something we say over and over at Ayehu – people don’t scale very well. 

Even the very best data center workers can only do so much. At some point, and that point is pretty much right now, end user self-service in concert with automation has got to take on a greater share of the service desk tasks all this growth in data is necessitating.

So when does it make the most sense for your end users to interface with Virtual Support Agents instead of live operators at the service desk? Well, one obvious answer to that might be as your organization’s first point of contact for L1 support issues.

We’re talking things like password resets, which Gartner estimates are responsible for as much as 40% of your service desk’s call volume.

Onboarding employees, and its closely related counterpart task offboarding employees, are also excellent tasks for a VSA.

How about VM provisioning and VM resizing?  These tasks lend themselves very well to a VSA interface.

Another good one is service restarts.  It makes a lot of sense to empower end-users to be able to do this themselves via a VSA.

And there are many, many more L1 support issues that would be great candidates for Virtual Support Agents.

But then what about your service desk?  What should your human agents do once the Virtual Support Agent has taken on all these tasks? 


Well, there are still probably a number of L1 support issues that are not well-suited for a VSA (at least not yet). The service desk can continue handling these.

Of course most, if not all of your L2 and L3 support issues are still best handled by a human service desk, for now.

And finally, vendor support issues are also probably still best managed by the service desk.

Still, deploying a Virtual Support Agent will clearly shift a lot of tedious, laborious tasks off of the service desk and free them up to do other things. But what kind of impact does this have on the cost structure of a service desk?

The average industry cost of handling an L1 support ticket is $22. In comparison, the average cost of a ticket handled by a Virtual Support Agent is just $2. Using these figures to do a back-of-the-envelope calculation of how many tickets can be off-loaded from the service desk to the VSA, will likely yield a significant reduction in support costs. What’s more, issues handled by VSAs often gets remediated much faster, resulting in greater end-user satisfaction than going through the service desk.

And if end users are going through a VSA instead of the service desk, then that means the VSA can reroute a trainload of L1 incident tickets away from the service desk, freeing up that staff to focus on more important, and more strategic things.  This is a huge value proposition!

What about the benefits VSAs provide to end-users?

Here’s an obvious one. Ask any end-user how much they love waiting on hold when they need an issue resolved, but no one’s available to help them. Waiting on hold can really degrade the user’s experience and perception of the service desk. With Virtual Support Agents, on the other hand, no one ever gets put on hold because the VSA is available 24/7/365. VSAs never take a break, or call in sick, or get temperamental due to mood swings. They’re always available and ready to perform on-demand.

Then there’s the mean-time-to-resolution metric, better known as MTTR. As every service desk knows, the speediness of their incident remediation outcomes is one of the major KPI’s they’re judged on.  Well when it comes to speedy MTTR, a VSA should be faster than a human being just about every single time.

Finally, does your enterprise have younger employees, particularly from the millennial generation? Of course it does! Just about every organization does, and some have them in far greater percentages than others. Well, guess what? This generation, raised on Facebook and mobile apps, generally prefers interfacing with technology as opposed to people. And that’s something we all know empirically because we’ve seen it for ourselves.

In addition to that visual proof, a survey conducted in 2018 by Acquire.io found that 40% of millennials said that they chat with chatbots on a daily basis! So, providing VSAs that empower younger workers with self-service capabilities might just give your organization a competitive advantage in attracting the best and brightest young talent to your company.

Questions & Answers

Q:          What are the pros and cons of using general purpose bot engines compared to your solution?

A:           General purpose bot engines won’t actually perform the actions on your infrastructure, devices, monitoring tools, business applications, etc. All they could really do is ingest a request. By contrast, Ayehu could not only ingest a request, but actually execute the necessary actions needed to fulfill that request. This adds a virtual operator to your environment that’s available 24x7x365. Additionally, Ayehu is a vendor-agnostic tool that is capable of interfacing with MS-Teams, Skype, etc. to provide a general purpose chat tool with intelligent automation capabilities.

Q:          Do you have on-premise solution?

A:           Yes.  Ayehu can be installed on-premise, on a public or private cloud, or in a hybrid combination of all three.

Q:          Do you have voice integration?

A:           Ayehu integrates with Amazon Alexa, and now also offers Angie™, a voice-enabled Intelligent Virtual Support Agent specifically designed for IT Service Desks.

Q:          If a user selects a wrong choice (clicks the wrong button) how does he or she fix it?

A:           It depends on how the workflow is designed. Breakpoints can be inserted in the workflow to ask the endpoint user to confirm their selection, or go back to reselect.  Ayehu also offers error-handling mechanisms within the workflow itself.

Q:          Does Ayehu provide orchestration capabilities or do you rely on a 3rd party orchestration tool?

A:           Ayehu IS an enterprise-grade orchestration tool, offering over 500 pre-built, platform-specific activities that allow you to orchestrate multi-platform workflows from a single pane of glass.

Q:          Can you please share the slide on IVA vs. ServiceDesk and elaborate a bit on the use cases?

A:           The entire PowerPoint file presented in this webinar can be found on SlideShare.

Q:          Can you explain in a bit more detail on intent-based interactions?

A:           Intent is just that: what the user’s intent is when interacting with the Virtual Support Agent (VSA). For example, if a user types “change my password”, the intent could be categorized as “password reset”. That would then automatically trigger the “password reset” workflow.

Q:          Can we use machine learning from an external source, train our model, and let Ayehu query our external source for additional information?

A:           Yes. Ayehu can integrate with any external source or application, especially when it has an API for us to interface with.

Q:          Can I create new automations to my in-house applications?

A:           Yes. Ayehu can integrate with any application bi-directionally. Once integrated with your in-house applications, Ayehu can execute automated actions upon them.

Q:          Is there an auto form-filling feature that can fill in a form in an existing web application?

A:           Yes. Ayehu provides a self-service capability that will enable this.

Q:          How can I improve or check how my workflows are working and helping my employees to resolve their issues?

A:           Ayehu provides an audit trail and reporting that provides visibility into workflow performance. Additionally, reports are available on time saved, ROI, MTTR, etc. that can quantify the benefits of those workflows.

Q:          What happens when your VSA cannot help the end-user?

A:           The workflow behind the VSA can be configured to escalate to a live support agent.

Q:          If there is a long list of choices, what options do you have? Dropdown?

A:           In addition to the buttons, dropdowns will be provided soon in Slack as well.

Q:          Did I understand correctly, an admin will need to create the questions and button responses? If so, is this a scripted Virtual Agent to manage routine questions?

A:           Ayehu is scriptless and codeless. The workflow behind the VSA is configured to mimic the actions of a live support agent, which requires you to pre-configure the questions and expected answers in a deterministic manner.

Q:          is NLP/NLU dependent on an IBM Watson to understand intent?

A:           Yes, and soon Ayehu will be providing its own NLP/NLU services.

Q:          Are you using machine learning for creating the conversations? Or do we have to use intents and entities along with the dialogs?

A:           Yes, you currently have to use intents and entities, but our road map includes using machine learning to provide suggestions that will improve the dialogs.

Q:          What are the other platforms from which I can deploy the VSA, apart from Slack?

A:           Microsoft Teams, Amazon Alexa, ServiceNow, ConnectNow, LogMeIn, and any other chatbot using APIs.

Missed the live Webinar? Watch it on-demand and see the above in action by clicking here.

New call-to-action