TESTING JEV ON LEAST-PRIVILEGE ACCESS DECISIONS

At Trustle, we manage IAM access for people and agents across cloud and SaaS platforms to achieve least privilege, giving users only the access they need. While LLMs help us improve security insights and navigate IAM complexity, they also bring significant drawbacks.

• They’re expensive at scale. In our role-selection evals, Gemini 3.8-flash cost about $3.08 per 1,000 decisions. That is small for one request, but it adds up when you make thousands of decisions a day.
• They’re slow. The same evals took 3.4 seconds per decision at the median, and 11.2 seconds at the 95th percentile. This creates a frustrating user experience when users have to wait for an answer.
• They can be inconsistent, and they can hallucinate. Asked the same question three times, Gemini changed its answer on about 6% of requests. Our assistant's eval suite even tracks a dedicated failure mode for invented role names.

While we have mitigated these drawbacks where possible, limits remain. Consequently, we were excited by TypeSafe AI's launch of Jev, a fast, low-cost "System One" decision model. We evaluated whether Jev delivers the accuracy needed for our access decisions.

Testing Jev for least privilege analysis

For an LLM, you pass it a prompt, and it generates text as an output. You can hold LLMs to a JSON schema, but the answer is still generated text, and any confidence the model gives you is something it wrote, not something actually measured. 

For example, a prompt might look something like:

You help users request cloud access with least privilege. Pick the role from the candidates below that lets the user do exactly what they asked, with the least access beyond that. Answer with JSON: {"role": "<role name or none>"}.

Request: I need to read the value of the DB_PASSWORD secret in project creds-stg.

Candidates:
- roles/secretmanager.secretAccessor: Reads secret values (3 permissions).
- roles/secretmanager.viewer: Sees secret metadata, not values (11 permissions).
- roles/iam.securityReviewer: Reads IAM policies everywhere (2,563 permissions).

Unlike LLMs, Jev evaluates a given state against defined, typed questions and limits choices to preset options to prevent role invention. Because each answer includes probabilities, you can configure decision thresholds programmatically. 

The equivalent Jev request is as follows:

{
"model": "jev-1.14.0",
"state": {
"request": "I need to read the value of the DB_PASSWORD secret in project creds-stg."
 },
  "questions": {
    "role": {
      "type": "choice",
      "instructions": "Pick the role that lets the user do exactly what request asks, with the least extra access.",
      "criteria": {
        "roles/secretmanager.secretAccessor": "Reads secret values (3 permissions).",
        "roles/secretmanager.viewer": "Sees secret metadata, not values (11 permissions).",
        "roles/iam.securityReviewer": "Reads IAM policies everywhere (2,563 permissions).",
        "none_of_these": "No listed role lets the user do what they asked."
      }
    }
  }
}

With the response being:

{
  "model": "jev-1.14.0",
  "answers": {
    "role": {
      "type": "choice",
      "choice": "roles/secretmanager.secretAccessor",
      "confidence": 0.99,
      "probabilities": {
        "roles/secretmanager.secretAccessor": 0.99,
        "roles/secretmanager.viewer": 0.0,
        "roles/iam.securityReviewer": 0.0,
        "none_of_these": 0.01
      }
    }
  },
  "usage": {
    "input_tokens": 433,
    "output_tokens": 79
  }
}

Jev picked the 3-permission role with 0.99 confidence. For a similar request, our existing search ranking had put the 2,563-permission iam.securityReviewer first.

During testing, Jev performed worse on broad questions like "How risky is this role assignment?" but excelled when evaluating narrow, focused signals combined programmatically. Similarly, Jev scored 62.6% when asked "is this phishing?", but reached 95.0% when split into five targeted signals and processed via logistic regression. (Beri: "TypeSafe's Jev scores 62.6% asked once and 95% split five ways").

Our risk scoring went through a similar arc:

v1–v4 ran on about 4,300 role assignments in our internal staging org. v5 ran on 110 public Google Cloud roles.

We also integrated Jev to suggest roles when users ask our agent for help. With proper context on tasks and permissions, Jev quickly identifies the ideal role. When requests become ambiguous, Jev signals low confidence, allowing us to fall back to an LLM effectively.

We tested this on 66 access requests with answer keys we wrote and froze before running anything. Requests included vague queries needing follow-up, dual-role needs, and misleading candidate roles. Evaluated on identical role lists across three runs per model:

While Jev alone was less accurate than Gemini, its confidence scoring was reliable: at ≥0.9 confidence, Jev was 96% accurate. Using Jev for high-confidence decisions and falling back to Gemini for others matched Gemini's overall accuracy, reduced over-broad role selection, cut Gemini requests by over half, and delivered sub-second responses for most queries. A pre-set 0.9 threshold achieved 97.9% accuracy on held-out test cases, outperforming Gemini's 97.2%.

Jev proved ineffective for:

• Rule-based scenarios where models add no value.
• Indirect privilege escalation (e.g., Cloud Build Editor running code via privileged service accounts).
• Counting tasks (e.g., misjudging if 21 permissions exceeded 20).
• Multi-role requests.

Jev's performance relies heavily on precise prompting: framing narrow questions, adding a "none of these" option, and replacing numbers with labels significantly improved results. When properly configured, decision models have strong potential for security applications.