an agentic AI benchmark, already built

Agentic AI, according to Teaganne Finn and Amanda Downie at IBM, is some kind of an AI system that can go do a task for you.

Agentic AI describes AI systems that are designed to autonomously make decisions and act, with the ability to pursue complex goals with limited supervision…. This type of AI acts autonomously to achieve a goal by using technologies like natural language processing (NLPs), machine learning, reinforcement learning and knowledge representation…. Agentic AI can adapt to different or changing situations and has “agency” to make decisions based on context. It is used in various applications that can benefit from independent operation, such as robotics, complex analysis, and virtual assistants.

Sounds like it has a lot of potential. Lots of complex goals out there that it would be great to have the AI go try to achieve. Tag this one #hugeIfTrue. But how do you figure out if an agentic AI stack is ready to depend on? Some suggested uses for agentic AI are customer support, health care, and supply chain operations. Testing the AI on that stuff means either you have to make a simulated environment and the quality of the results are only as good as the simulation that the AI is working in, or you test on real work with all the risks that come with that.

What we need, in order to figure out if an agentic AI is ready to start working on high-stakes tasks, is some kind of a task that has

• realistic level of complexity: require a mix of information processing tasks such as completing forms and attaching appropriate files.

• low cost of failure: expect that early agentic AI will fail, and that should be fine. Don’t wait to test new systems that are promising but lack a record of success, or require someone to do constant manual checks of the AI’s work.

• easy to measure success: some tasks are harder to evaluate than to do. In order to make an agentic AI benchmark practical, we should be able to write a simple pass/fail script.

What kind of task is that? Most complicated tasks are too high-stakes to make a good benchmark, and most low-stakes tasks are too simple to make a good benchmark. And complex tasks tend to be hard to evaluate. Where should an agentic AI evaluation project look?

Good news. Such a task does exist. As the California Privacy Protection Agency decision in a recent online retailer case makes clear, the process for people to do a Right to Know (RtK) under California law is often over-complicated. Companies tend to add extra steps. All that malarkey is a big hassle from the human user point of view, but considered as an agentic AI test suite, it’s buried treasure. If you want to test agentic AI systems you don’t need to go write a bunch of weird corner-case web forms and bureaucratic processes—the test suite has been written for you by all the developers of weird and wonderful RtK mazes. And the results of an RtK are straightforward to check with a script. Yes, the data is largely crap but the test harness can do some pattern matching and spot that you’re getting about the crap that you expect.

The end result will probably be that companies make RtK processes more straightforward, to avoid enforcement actions and the costs of processing agentic AI traffic, but for now we have a free test suite for agentic AI. Let’s see how well the different agentic AI tools do. More: encouraging first result from an AI right to know

Related

personal AI in the rugpull economy

Bonus links

German court rules cookie banners must offer “reject all” button by Skye Jacobs on TechSpot. The Hanover Administrative Court ruled that websites must display a clearly visible “reject all” button on cookie banners if they offer an “accept all” option.

Vermont state legislators: Big Tech doesn’t want you to know what’s happening to your kids online. We do. The truth is, we built this bill to withstand court scrutiny. We studied rulings in other states. We tailored the language. We removed any provision that regulated speech and focused only on product design and data processing. We listened to civil liberties groups, constitutional scholars and children’s advocates.

Priced Out: How Surveillance Pricing Leaves People with Disabilities At Risk by Ariana Aboulafia and Nina DiSalvo. Many people with disabilities regularly purchase items related to their disability, and do not have much choice in what they need, or when they need it. They may also be limited in where they can shop due to mobility limitations, difficulty finding accessible transportation, or other factors related to their disability. This means that both the privacy and consumer protection concerns that apply to all consumers in the context of surveillance pricing are particularly worrisome for disabled people.

Trying to avoid a heart attack, Instagram attacked me with ads by Bob Sullivan. Meta/Facebook/Instagram knew exactly where I was and what I was doing. But it didn’t just show me relevant ads. It knew I’d be vulnerable. It knew I might get bad news. And then it targeted me with crazy, untested products that would probably make me sicker. It’s vile and it needs to stop. This isn’t capitalism and it isn’t free speech. It’s using technology to attack people when they are nearly defenseless.

European Defense Tech brought engineers together for the first time for a hackathon for the front lines in Ukraine by Mykhailo Liuksikov. When you don’t understand the issues, when you don’t talk to people who have been on the front lines and understand the problems, that is, what is really important, that is why all these hackers need feedback on what they are doing. — Larysa Visengeriyeva, co-organizer of the European Defense Tech Hackathon.