Questions every CTO should ask a software development partner about AI

In 2026, every development partner claims AI capability. It tends to appear as a line on their website, a mention of Copilot in a capabilities deck, occasionally a case study involving a chatbot.

The questions that separate genuine AI integration from a marketing claim are operational. Most buyers never ask them. What follows is a framework for the ones that should. Use it before your next evaluation conversation. The answers will tell you more than any portfolio.

The six questions

Question 1: Which AI coding tools does your team use daily and which do they not use?

The second half of this question matters as much as the first.

A team with genuine AI integration has opinions. They can tell you which tools are in their daily workflow, which they evaluated and retired and why. They know the difference between what Cursor does well and what it does not. They have made deliberate choices based on real experience.

A team without it will tell you they use the latest AI tools and are always exploring new options. That is a positioning statement, not a process description.

What a strong answer sounds like: Specific tool names with specific use cases for each. A clear view on what works and what does not. Evidence of evaluation rather than adoption.

What to listen for: Whether the answer is grounded in real operational experience or stays at the level of general positioning. The difference between a process description and a marketing statement is usually audible within the first two or three sentences.

Question 2: What is your human-in-the-loop review process for AI-generated code?

AI coding tools produce output that is subtly wrong in specific and well-documented ways. Inconsistent architecture, security vulnerabilities, open-source licence contamination, logic errors that pass basic tests and fail under real conditions.

A credible partner has a documented process for catching these before they reach production. Human review of all AI-generated code, automated licence scanning in the CI/CD pipeline, security testing that specifically accounts for AI-generated output.

What a strong answer sounds like: A described review process with specific checkpoints, a clear understanding of the failure modes and evidence that quality standards apply equally to AI-generated and human-written code.

What to listen for: How specific the answer is about where in the process AI-generated code gets reviewed and by whom. A general commitment to quality is not the same as a described process with checkpoints.

Question 3: How has AI tooling changed your onboarding speed into a new codebase?

This question tests whether AI integration has produced a measurable operational change.

Onboarding into an unfamiliar codebase is one of the most time-consuming parts of any new engagement. A team that has genuinely embedded AI into how it works should be able to describe specifically how that process has changed. What used to take two weeks, what takes less time now, and why.

If the team cannot answer this with any specificity, AI has not been integrated at the operational level where it would make a difference to your timeline. 

What a strong answer sounds like: A specific description of how onboarding has changed, what tooling is used at that stage and a realistic indication of the time difference on a typical engagement.

What to listen out for: Whether the answer describes an actual change in how the team works or stays at the level of general capability claims. A team that has genuinely integrated AI into onboarding will be able to describe what that looks like in practice, not just assert that it is faster.

Question 4: Can you share before and after delivery metrics from a project where AI was a factor?

This is the question that reveals whether AI capability has been measured or simply assumed.

Claiming AI capability is straightforward. Demonstrating its operational impact with actual numbers is a different thing entirely. A team with genuine AI integration tracks delivery metrics. They know how code review cycle times have changed. They can point to a specific project and describe what was different because of how the team worked.

What a strong answer sounds like: Specific metrics from a real engagement, even approximate ones. Evidence from the team's own delivery rather than a vendor claim about AI productivity in general.

What to listen for: Whether the partner can connect AI tooling to a specific project outcome or stays at the level of general productivity assertions. Confidentiality is a legitimate reason not to name a client. It is not a reason to have no metrics at all.

Question 5: How do you handle AI-generated code in regulated environments or where IP ownership matters?

This question matters for any SaaS business in a regulated sector or where IP ownership is commercially sensitive.

AI-generated code raises specific ownership questions that are not always addressed in standard development contracts. A credible partner has clear contractual terms covering AI-generated code explicitly, a considered position on how AI tooling is used in regulated environments and an understanding of the difference between proprietary code and underlying libraries.

What a strong answer sounds like: Clear IP terms that explicitly address AI-generated code, a described approach to regulated environments and evidence that the team has thought about this rather than defaulting to a generic contract.

What to listen for: Whether the partner has a considered position on this or is encountering the question for the first time. Either is possible. What matters is whether they engage with the substance of it or deflect.

Question 6: What does your team do when an AI tool produces confident but incorrect output?

This is the question that reveals operational maturity.

AI coding tools produce incorrect output with confidence. It is one of the most well-documented failure modes of current tooling. A team that has worked with these tools at scale has encountered this and has a process for catching it.

What a strong answer sounds like: A specific description of how incorrect AI output is caught, the safeguards in the review process and ideally a real example of where this happened and how it was handled.

What to listen for: Whether the answer reflects genuine experience of this failure mode or describes a general code review process that does not specifically account for it. The teams that have been using AI seriously at scale will have encountered this. The ones that have not will give a more generic answer.

How to use this framework

These six questions are not a gotcha. They are a way of giving a development partner the opportunity to demonstrate operational maturity rather than just claim it.

Run them in any evaluation conversation. The partners who answer with specificity are demonstrating that AI is genuinely part of how they work. The ones who answer in generalities are demonstrating that it is part of how they market themselves.

It is also worth considering a short discovery phase with your preferred partner before committing to a full build. It lets you observe how the team actually works rather than relying on how they describe it.

Frequently asked questions

What questions should I ask a software development partner about AI in 2026? The six most useful questions are: which AI coding tools does your team use daily and which do they not use; what is your human in the loop review process for AI-generated code; how has AI tooling changed your onboarding speed; can you share delivery metrics from a project where AI was a factor; how do you handle AI-generated code in regulated environments; and what does your team do when an AI tool produces confident but incorrect output.

How do I evaluate whether a development agency genuinely uses AI? Ask for specifics rather than accepting general claims. A team with genuine AI integration can name the tools they use, describe the use cases for each, explain their review process for AI-generated code and provide at least approximate metrics on how AI has affected their delivery. Vague answers about using the latest tools are a signal that AI has not been operationally embedded.

What is the difference between a development partner that claims AI capability and one that actually has it? The difference is operational specificity. A partner with genuine AI integration has a documented review process for AI-generated code, tracks delivery metrics that reflect AI's impact and has a considered position on IP ownership and regulated environments.

How should AI-generated code be reviewed before going to production? Credible development partners have a documented review process that includes human review of all AI-generated code before it reaches production, automated licence scanning in the CI/CD pipeline and a defined approach to security testing that accounts for the specific failure modes of AI-generated output.

How Aecor helps

If you are evaluating development partners and want to understand exactly how we work, which AI tools we use and what genuine AI integration looks like in practice across a real project, a free 60-minute technical discovery call is a good place to start. You can also read more about how we approach AI Automation as part of our engineering practice and see how it has played out across real projects in our case studies.

Book your free Technical Discovery Call