Webinars
Apps in ChatGPT with Simon B. Støvring, Principal iOS Engineer at Framna
6,306 views
This segment, led by Simon B. Støvring, Principal iOS Engineer at Framna, explores how companies can now appear inside ChatGPT with fully branded, AI-powered experiences. Real examples show how early adopters are already using the ChatGPT App SDK, Agentic Commerce Protocol, and ChatKit to create new touchpoints within OpenAI’s platform.
It also includes a first look at how ChatGPT apps are built, how they connect to your existing infrastructure through the MCP server, and why this emerging channel is becoming a meaningful part of the digital product and customer-journey landscape. For teams working with modern digital products, this is a shift that cannot be ignored.
View transcript
So let's have a look at what apps in ChatGPT are, and how we can build them. But we'll start by addressing the elephant in the room here. Because what we'll be discussing today, or what we are discussing today, is currently only available in the US. Now, OpenAI has said that they expect to bring apps in ChatGPT to EU users soon. And with OpenAI or AI in general moving so fast, we expect that to be really soon. And we're already working with some of our partners on prototyping and building apps for ChatGPT. And we think that even though it's not available in the EU yet, this is the time to get started. So you're ready when apps are available in the EU. Now, back in October, OpenAI had their dev day keynote, where they launched apps in ChatGPT, along with the apps SDK to build those apps. Apps in ChatGPT offer companies and developers or enable companies and developers to offer their services through voice and chat, along with interactive UI in line within ChatGPT. In this recording, we see an example of an app on a ChatGPT app where we use Spotify. It's Spotify's ChatGPT app. Spotify is one of the launch partners in the US alongside booking.com, Zillow and Canva. And this Spotify ChatGPT app enables us to interact with our Spotify account from ChatGPT. So we use Spotify here or ChatGPT to create a playlist with tracks from Ed and Taylor. And then we ask ChatGPT to find a few surprises from other artists to put on the playlist. And notice that even though we don't specify the last names of these artists, Ed Sheeran and Taylor Swift, ChatGPT and Spotify automatically knows what Ed and what Taylor we're referring to. Now, in this case, we just accept the playlist created by ChatGPT as it is, but we could choose to refine it. For example, we could choose to have Spotify create a playlist with only acoustic tracks from the artists. So we could refine this playlist through chat with ChatGPT. And once we have the playlist ready, we can open it directly in Spotify and start playback. Now, in this example, we pull the Spotify app into the chat by typing slash Spotify, which kind of launches the app in ChatGPT. We could also do that by pressing the plus and then browse and find the Spotify app within ChatGPT. But ChatGPT can also suggest apps based on the content of the dialogue, based on the conversation we're having in ChatGPT. Previously, we saw the mobile app, ChatGPT's mobile app. In this case, we see the ChatGPT website, that is ChatGPT.com. And in this example, we don't explicitly pull in the Spotify app, but rather we talk about the creation of a playlist. And then ChatGPT recommends us to use the Spotify app. You can see that in the bottom of the dialogue here. As a user, we can then approve the use of Spotify's app, sign into our account and then create a playlist similar to what we saw earlier. All based on the recommendation of ChatGPT. So ChatGPT will constantly monitor our conversation and suggest relevant apps. And of course, these examples can be more powerful for other conversations. For example, imagine outlining your travel plans and using ChatGPT to do so. And then it recommends apps that are relevant for booking your entire journey. It could be flight tickets, Airbnb, and so on. And that's actually what we're seeing Booking.com doing with their app. Now, what we're seeing here with the introduction of apps in ChatGPT are the pieces coming together to form a platform. OpenAI is going from offering ChatGPT as a standalone product to offering it as a platform that companies and developers can build upon. OpenAI has previously offered plugins and connectors. But they really never gained the traction that OpenAI might have hoped for. And these are now deprecated in favor of the way more powerful apps in ChatGPT that enables these interactive UIs along with conversational interactions. And this interactive UI is really the kicker here. Now, some of the reasons we believe that the pieces are coming together to form a platform. Some of those are listed here. These are key in having a good platform. We can now offer branded and interactive UIs directly in ChatGPT. And these can be really versatile. OpenAI has announced that there will be a dedicated directory, public directory, listing the apps that are available in OpenAI. And users can browse and search this directory of apps. That's not really unlike the app store that really made the iPhone and the Android platform to build upon. OpenAI will be promoting apps that live up to their standards. And they will be promoting them even heavier if they are featuring them even more prominently if they exceed their standards. And that could be, for example, as we saw within conversations. OpenAI has also announced that they will enable companies and developers to monetize their apps from within ChatGPT. And as we'll see later, or get into more in depth later, OpenAI has also announced that they will be launching their own hardware. All of these are key ingredients in having a successful platform for companies to build upon. Actually, we've been diving deep into the apps SDK that enables us to build apps in ChatGPT since it was launched back in October. And actually, we've had a handful of people working almost full-time on building and designing experiences for apps in ChatGPT. Just 24 hours after the launch of apps in ChatGPT back in October, we had an early prototype ready that lets us control our Philips Hue lights directly from within ChatGPT. This is all built on Philips Hue's existing public API. So it's built on their existing infrastructure. And in this example, we see that how we can ask ChatGPT to change the state of our lights. For example, to turn them on, turn them off, change their colors, and we can even control entire rooms at once in one query. And I should mention that the prototype in this video runs a bit slow, because as you remember, these apps are not available in the EU yet. So everything goes through VPNs. But we're confident that this will be much faster once it's deployed to production. We think this is a great example of how we can combine new technology Apps in ChatGPT with some existing technology. These light bulbs. And it kind of connected the two worlds We've started partnering with a handful of the user partners to explore how their product can be used or brought into ChatGPT. For example, we have Praktikertjänst, which makes it easy for patients to book urgent dental appointments. So patients can describe their dental issue, get treatment recommendations directly from within ChatGPT. And ChatGPT can then provide some possible time slots for the patients to book. The patient can select a time slot that's convenient for them. And upon selecting a time slot, users are redirected to the app to complete the booking flow. From working with some of our partners on building ChatGPT apps, we found that this will likely be a common pattern, at least in the early days of ChatGPT. And what I'm talking about here is the pattern of starting a conversation in ChatGPT and then transitioning. When you go into a more complex flow, then transitioning it to an existing web app or an existing app, native app or a mobile app. Starting an interaction in ChatGPT and then handing off to an app or a website enables us to utilize ChatGPT the way that it, for what it's actually best at, browsing, refining and understanding data. And then offload longer and more tedious flows to existing products, the web or the app. This is also a great way for companies to get started with ChatGPT without a huge investment or a great way to get started building apps for ChatGPT, I should say, without a huge investment. As we can identify key interactions, bring them into ChatGPT and then reuse part of our existing infrastructure, part of our existing products to kind of complete the flow. Now, let's just have a brief look at how these apps in ChatGPT are built and how they can fit into your infrastructure. ChatGPT apps are built using model context protocol. So this is essentially OpenAI commercializing model context protocol and then expanding it, this protocol, with interactive UIs. And actually just last Friday, Anthropic and OpenAI announced that these interactive UIs will be embedded into the model context protocol, embedded into the model context protocol, the protocol will be expanded to support this, and it will be a new open standard that other companies can adopt. So by building an app for ChatGPT, you're not really tied to that platform, you're not logging in, your product will become portable and supported by other vendors as well. Now, as I mentioned, model context protocol is an open source standard for connecting AI applications to external systems. So we can think a bit of it like USB-C. Were USB-C enables us to connect different hardware products for example or MacBook and our iPhone MCP enables us to connect AI application like ChatGPT to external systems like data sources or tools that can perform operations, modify, and read data. An MCP server that is a system communicating over MCP plays a key role in building a ChatGPT app. It is essentially the ChatGPT app. This shows a simplified overview of an infrastructure that might look like what you already have today. In this case, it's a mobile app communicating with a backend which in turn communicates with some underlying systems for sending push notifications, reading data, and so on. Now, for most projects, an MCP server, and as a result, the ChatGPT app fits naturally into an existing infrastructure to serve as a new channel. The MCP server can typically be connected to an existing backend and utilize the data already available there and the operations already available there, and then sit between your backend, your existing backend, and ChatGPT, which becomes the frontend. So the ChatGPT app becomes a new channel that we plug in to the existing infrastructure and that can run in parallel with your existing mobile app or website and actually utilize, reuse some of your existing logic. These MCP servers are HTTP services that can be built using any programming language, but Python and TypeScript are among the popular choices. They've become quite popular in the AI world, and some of these vendors, including OpenAI, are providing developers with SDKs, Python and TypeScript SDKs, that they can utilize to build their MCP server and, in turn, their ChatGPT app. There are different ways that this MCP server can fit into your infrastructure. For example, your existing backend can expose this MCP server. It can be built directly into your existing backend and deployed in your existing infrastructure. Or it can be a completely separate service that's deployed elsewhere. Now, the MCP server can also, can not only be utilized to integrate or build a ChatGPT app and integrate your service and your offerings into ChatGPT, but over time, you could also imagine your existing web app or your existing mobile app utilizing this MCP server for doing some clever AI things. The interactive UI components that play a key role in ChatGPT apps are built with HTML, JavaScript, and CSS, and it's quite common to use React to kind of bundle those together and build upon. Now, while MCP servers are the foundation of ChatGPT apps, there's a bit more to it than that. OpenAI is offering a toolbox that enables us to provide some quite complex operations within ChatGPT. For example, we can perform navigation in response to user actions to navigate within our interactive UI and show different UIs based on user interactions. We can also trigger server-side actions. For example, if a user clicks a button in the chat, in our widget or in our interactive UI, we can send an action to our server to manipulate data. We can even send conversational follow-ups based on user interactions. These conversational follow-ups will be reflected in the conversation as a message sent by the user and ChatGPT will then respond to it. There are also ways to request alternative layouts. So by default, these widgets, these UIs are embedded or in line within the conversation. But users can also take them into full screen or they can run them picture-in-picture so they can interact with the app or the UI and the conversation at the same time and kind of interact with them independently. So after experimenting with ChatGPT apps over the past couple of months, we've formed some opinions on how we best design these. And I just want to take you through a couple of the noteworthy ones. And most notably, we've found that it's important to embrace the chat as the primary way to navigate and interact with widgets. I already covered that a bit. We should avoid building complex flows in the widgets. And instead, we should hand off to the website or app when flows become long or tedious. For example, to gather data about a user, like a sign-off form. OpenAI or ChatGPT is not really suited for that, but your existing web app or your existing native app is probably well suited for that. We also found that to make the most of combining chat with inline UI, we should explore how ChatGPT's picture-in-picture can be utilized to enrich the user experience. So picture-in-picture is a quite exciting part of ChatGPT apps, because using that lets users navigate the app with voice and chat, while seeing a continuously updated UI as their conversation changes. So our UI can reflect what's going on in the dialogue. We've also found that while we're technically free to design the UI in any way we'd like, OpenAI has some quite strict guidelines on how they want these apps to be designed. And we have yet to see how OpenAI is going to enforce these guidelines and how much leeway they're actually going to give us. But what we're seeing now is that based when we're interpreting these guidelines, we have a bit less room for branding our apps than we're used to from web and mobile apps. And it's worth noting here that OpenAI will evaluate apps in detail. And they may feature any app that exceeds their expectations. So designing a proper app is really key here. Just in order to stress how serious OpenAI is about the direction they're heading into with the introduction of ChatGPT apps, they've publicly said that they're working on a family of AI devices. They've partnered with Johnny Ive, former chief design officer at Apple, and instrumental in defining the current hardware lineup that defines modern Apple. OpenAI has said that their initial device in this family of devices will serve as a third core device will serve as a third core device. That is, they expect us to bring it with us together with our laptop and our phone, and then they will serve as the third core device. They've said that this third core device will be unobtrusive and fully aware of the user's life and surroundings. So it understands what we're doing, what we have done, where we're heading. OpenAI has said that they're aiming to unveil the first of these devices in 2026. and launch it in 2027. So it's just around the corner. And they expect to be, they expect the first device to be the fastest device to ever sell 100 million copies, 100 million devices. So OpenAI has big plans, and there's no doubt that ChatGPT apps and ChatGPT as a platform will play as an instrumental role in the launch of this hardware. And speaking of hardware, it should be noted that the future of chat and voice driven interfaces might not be as far into the future as we could think. In September, Meta launched a new lineup of Meta Ray-Ban AI glasses, along with the Meta Wearable Devices Access Toolkit that enables developers and companies to build experiences on their, or for their AI devices. OpenAI has also stated that they're exploring glasses as part of their family of AI devices. So if we want a glimpse of the future, we might not need to look further than some of the hardware that's already on the market. And I don't think it takes much imagination to see how a product like Meta's AI glasses and apps and ChatGPT can be combined to create a category-defining product. And with that, I'll hand it off to Michael, who will take you through how this changes the customer journey.