Why Observability Decides AI in Production | Built to Scale: Secure AI Factory Ep. 2 (Observability)

Rajeev Khanolkar

Here is how I look at it. Having spent years around network management, fault management and performance management, when I look at secure a Factory, three things come to my mind. One is performance. Second one is security and the third one is token mix. I think in terms of performance, if AI or LMS are going to be part of your reasoning infrastructure, application infrastructure, you want response in a decent amount of time, right?

That’s why performance is important. My infrastructure giving me the right performance. Security. Why it is important. Security of LMS. The security of applications is very important because you do not know whether it has been poisoned. You just want to know are there any connections to your LMS or applications or AI infrastructure that is deviating from the standard security that is going to impact the results?

And last one is token omics. We all are saying that if you go to any of these public websites, whether it is cloud or ChatGPT or other environment, you start using tokens and you don’t know when you’re finished using them and you get a huge bill. And that’s why having Splunk dashboard, it’s very valuable. So there are at least you can compare ROI when you implement on prem security a factory.

Matt Locknane

And you look at the customers who are doing AI in cloud. Right. Because it’s easy to develop there and scalable, but the costs are prohibitive. And so when you look at the actual token cost of cloud versus on prem, you could actually figure out what is your savings.

Rajeev Khanolkar

I agree, I agree with this. I never thought that some parameter like this would be so critical to manage an infrastructure which is not application, not network performance. It’s a token consumption and it is hard for normal network and infrastructure administrators of the past. But if you work in the cloud, you don’t know when you are consumed it and how much you have consumed it.

Matt Locknane

We’ve seen instances where customers have public facing AI chatbots that are being used to do whatever the customer wants as they dial in, right?

For example, I can go to a customer insurance bot and I can say, hey, write me some code that does x, y, z, and if the proper guardrails aren’t put in place, it’s burning tokens to go ahead and do that now. Most of those are cloud based today. That’s a lot of money that’s being spent on work that doesn’t have anything to do with increasing your particular customer base or customer satisfaction.

Rajeev Khanolkar

And the thing about tokens is, you know, when you upload a file into cloud, you know, hey, I consume so many gigabytes of space and we when you extract cloud providers used to charge you right, for the amount of data you extracted. At least you know what you’re taking with this token. You don’t know what the hell is happening, right? It’s totally invisible. It feels like somebody just robbed me, right?

Matt Locknane

Yeah, I saw I saw a post the other day that somebody used, like, 400,000 tokens in the space of five minutes, scraping bad data that they didn’t even realize the AI agent was scraping.

The nice thing about this piece is the dashboards cover everything around the physical hardware, the AI workloads, the Rag implementation. You have all of that at your fingertips from the single dashboard.

One of the key topics in secure AI factory is the visibility piece that this block Observability Cloud brings. So what is observable mean to you with your background coming from the security side?

Rajeev Khanolkar

When I look at observed ability, I want to know when you look at any security attacks there are only two attacks, two types. One is denial of service. Whether somebody identity is compromised, every single security instance is either one or the other. Let us assume that why Splunk Observability can give you insight into hey. Am I subject to a denial of service attack? Like, somebody is taking away a lot of TCP connections which are available to connect to my application or application server, right?

That’s one part. The other part is, hey, what is the chances? Where is my identity being? Which areas of the application or infrastructure has it touched and why has it touched it? When a real user is not supposed to be there, from that perspective, it is very important to see how your identity has traveled through the journey.

And this is what I look at observability for.