AI Experimentation Without Governance Is Invisible, Unaccountable and Impossible to Scale
I gave six AI experiments portfolio-level visibility. Here's what happened next.
Our VP wanted to understand what we were doing with AI so Program Management compiled a list from Product and Engineering across our area’s four domains: Design, Product Creation, Merchandising, and Innovation.
The list was interesting, but it was not what he wanted.
No one in my leadership chain really wanted to deal with this, so they gave me the list of known AI experiments and asked me to run it as a portfolio.
I had asked for more work because I wanted a promotion, and thought, sometimes, it’s good to do the thing no one wants to do. I said sure, no problem.
That was the whole conversation.
In this post, I’ll walk you through my first 3-6 months of standing up the AI Portfolio which included assessing viability, defining status content and cadence, coaching product and engineering use case owners going through AI pilots and PoCs for the first time, and a few unexpected things I learned along the way.
Founder Field Notes: Practical lessons from leading enterprise programs, standing up AI initiatives, and helping organizations navigate new ways of working.
These posts connect real-world experiences to the operating models, governance approaches, and design principles explored throughout the lab.
Various Stages of Maybe
The AI experimentation list was a mixed bag: a SaaS vendor pilot for mood boards, a natural language query proof-of-concept for merchandising, and some co-pilots in various states of maybe, plus all sorts of metrics and various levels of the business either knowing about it through a demo or asking for something like it without having seen anything tangible.
Program runs execution. Product owns the roadmap. Portfolio works across both to enable leadership visibility and decision making.
When I followed up on the list, I noticed AI experimentation was happening outside of the normal channels.
No program resources. No roadmap visibility.
I wouldn’t call it shadow AI because it was all in various stages of maybe. Nothing in production.
What Should Status Look Like?
I worked with a very smart Senior Product Manager (SPM) to stand-up weekly status for the AI Portfolio.
Our VP was very hierarchical. All communication flowed between me to the SPM, then between the SPM to the VP.
The Senior Product Manager repeatedly tried to exit the loop and leave it to me.
Our devil didn’t wear Prada, but the expectations were similarly high and undefined.
We were under pressure to quickly do something without specifically knowing what that thing should look like. We were annoyingly incompetent at reading the VP’s mind.
Every week we restructured status reporting based on the previous week’s feedback around the following questions:
Who was accountable for the AI use case?
What were they trying to prove or provide to which stakeholders?
What was the next milestone?
Did they need help?
I also got permission to take an AI Portfolio course from MIT’s CSAIL department to work out the details as weekly homework assignments, and it honestly helped a lot. I was missing the perspective that the VP wanted:
where was all the experimentation leading in terms of business value,
what decisions did he need to make, and
how could he talk about this work to his boss who wanted to see more AI across technology?
We couldn’t seem to get it right, until we did.
After spending over 20 hours editing and iterating on a 1-page email status every Tuesday to Friday, the VP implied that we had achieved status nirvana, with several edits on wording that would make it ready for his boss to read.
We also had his permission to send the status email to his boss and our distribution list without his weekly feedback before we sent it.
“Are you sure? Maybe treat next week as a test?“ we asked.
The VP agreed, but he became impossible to pin down and never reviewed our status again before sending.
Needless to say, hitting send felt like Russian Roulette every week, then every other week, until the entire IT portfolio moved to a monthly review cadence as one cohesive portfolio view.
Once the reporting stabilized, something more interesting started happening.
The people doing the work and providing the status started to appreciate being visible and accountable.
In all my years of doing program management, absolutely no one likes status reporting, but they said it helped them stay accountable.
It is hard to stay accountable when you’re operating outside of all the normal IT functions as an AI experiment.
Visibility creates accountability.
Accountability eventually forces a harder question: is this real work with potential business value or just an interesting experiment?
Is It Integrated Into the Roadmap?
While status was getting sorted, I began to focus on two questions unrelated to status which would eventually help me integrate AI experimentation into the regular roadmaps that were managed, prioritized and reported by the people who were in charge of them:
Who knew about the work and agreed to it?
Is it a PoC or Pilot with a set timeline and metrics to drive a go/no-go decision?
Three of the use cases had actual visibility and sponsorship from the business with Product Managers and Engineers working on them:
SaaS vendor pilot for Footwear Design Mood Boards and Concept Design
Natural Language Query for Manufacturing
Co-Pilot for Footwear and Apparel Prototype Research
These had confluence pages, solution design documentation, and metrics. Plus, regular demos with the business partners who were intended to use them.
The other three use cases did not.
And, there wasn’t a clear definition of PoC vs Pilot across these various domains that rarely, if ever, worked together in the course of a regular roadmap outside of a few PI Planning cross-functional dependencies.
Proof of Concept (PoC) proves a hypothesis that a concept is technically feasible.
Pilot proves that it can be successfully integrated into production workflows and result in actual end-user adoption.
I wrote the definitions for PoC and Pilot and had the Senior Product Manager circulate them across Product Management so that I could use them to manage the AI experimentation portfolio.
Unlike my previous innovation office work, this division didn’t really experiment aside from an innovation sprint during PI Planning or maybe a hackathon run by central IT.
They needed the language and structure to help them focus on the right hypothesis.
And, most of the program managers in this space were over-seeing large, multi-year efforts and lacked the time, interest and expertise to right-size program management for smaller efforts.
So, I wrote my name down in the AI experiment register as the program manager for the four use cases without program support.
In one use case, an engineering manager was running the PoC without product support which resulted in several product-led conversations to shut it down or merge it with existing work. I said very little in those meetings. Product needed to sort it out.
All this documenting became critical when the scope of our status reporting stopped being within our department and started to become visible at the cross-IT level.
Do We Look Legit?
About eight months into managing the AI portfolio, I hear that IT was centralizing all reporting on a monthly basis.
This meant a few things:
The end of individual departments each sending their uniquely formatted weekly status to the EVP of Technology
Consolidated AI portfolio templates that handled PoC to Pilot to Scale reporting across all departments
The need to highlight our wins and tell our story at an executive level with confidence
AI experimentation truly integrated in existing roadmaps with Program, Product and Engineering owners that would be accountable for the work
A funny thing about this transition was that I had to fight to be in the portfolio meetings to prep for the monthly AI status.
Just as product and program accountability was written but not necessarily official, my role as portfolio was considered a stretch despite me doing all the work for months.
So after a few months, I handed off my role as the AI portfolio person to the regular portfolio person for our department.
I would not say there was excitement. More work is more work. Extra work that is partially integrated in one’s regular work is annoying.
Looking back, I didn't realize I was learning a pattern that would show up repeatedly in enterprise AI adoption.
Organizations struggle because they lack a way to make AI visible, accountable, and governable as they move from experimentation to production. It often starts outside the normal channels, especially when it comes from outside IT.
What I Miss About Running the AI Portfolio
A few things about this experience stand out to me:
The opportunity to truly work across a department within different domains and mindsets: innovation and merchandising think very differently
Seeing how others see AI as an opportunity and learning with them as the prove out the hypothesis
Creating operating models that drive clarity without unnecessary overhead
At the time, I thought I was standing up an AI experimentation portfolio.
Looking back, I was learning something much more useful: AI scales when ownership, accountability, governance, and visibility become part of the system surrounding it.
It wasn’t a clear cut role which means opportunity was negotiable and often earned through a series of wins to gain trust and organizational permission to be at the table where decisions are made and priorities are clarified.
The part about AI work happening outside the normal roadmap process stood out to me. I've seen plenty of excitement around the experiment itself.
Figuring out ownership and what success actually looks like is usually the difficult part i think.
Reading your article made me realize how the orgs still operate, and I felt it for you in that role so much, Judy! I’m glad you found it useful and took the lesson learned, but the bigger question is, did you get promoted after all this work?