Empowering Web Application Developers: Unleashing the Potential of Docker Desktop and Grafana Cloud

To start off this talk, we’re going to talk about verification development and how we can add observability to it.

A bit about myself — I am Shahzeb, and I work with Grafana Labs as a senior software developer. How many of you have used Grafana? Okay, good. I’ve been working with it for almost two years. Overall, I have over eight years of work experience among deployment, development, and integrations. And I have developed the Docker Desktop extension for a Grafana Cloud. You have been hearing a lot about Docker Desktop extensions. They are really cool. And we have developed an extension that lets you easily connect Docker Desktop to Grafana Cloud.

I have over five years of development experience as well, like developing in React, Node.js, and many other technologies. I am from Pakistan, which is around 8,000 miles away from here, around a 12-hour time difference from back home. It’s like at least midnight for me. I work remotely with Grafana. We are a remote-first company, and we have like 50-plus different countries working with us. And each country has a different culture. So, fun fact about my country, 90% of my family thinks I’m unemployed because I work from home. Because working remotely is not really a concept there. But we are getting there. Moving ahead, let’s go into the agenda.

Observability in web applications

We are going to talk about why observability is important in web applications. I’m going to show you my web application setup with Docker Desktop. We’re going to connect Docker Desktop with Grafana Cloud, enable some additional Grafana integrations through the extension. And then also see how we can load test this application with this combination of Grafana and Docker Desktop. At the end, we’re going to have a holistic view of how our whole environment is set up and what we can do more with it. So, quickly, this is not here to show you what’s coming up. I’m not going to talk about this right now. We’re going to build it piece by piece, step by step. But this is just like a preview of what’s coming up.

Why is observability important in web applications? I’m just going to quote some very popular, very widely used facts around the industry to show this importance. Google found a 0.5 second delay causes a 20% decrease in repeat traffic. And that also persisted when the people went away. Similarly, Amazon found that every 100 milliseconds slower the site loaded, they lost 1% in revenue. And their revenue, I’m pretty sure it’s billions right now. Walmart also found that based on their web page loading performance, their conversion rates vary a lot. All three of these are talking about web applications, Google search, Amazon, and Walmart. And they tell how significant it is to make sure that your application is always up and running and that it’s performing at its optimal level, which can be done through observability.

Generally, all over the world, how our application development workflow works is that we do design. We do development. We do testing. We do go live. And then once everything is live, a support manager comes in and thinks about how I’m going to observe this thing. This is not the way we should do it right now because of all the tools that we have. Our purpose is to move this observability part back into the development stage so that you have the tools that you can develop all your monitoring infrastructure alongside your application development as well. That’s basically what I’m going to show you today — how you can do that with Docker Desktop and the Grafana Cloud extension.

Demo

Quickly, our web application is a very simple web application. Running inside Docker Desktop, I have a React frontend, a Go backend, MySQL database, and it’s all being served through an Nginx proxy. Let me just quickly show you in terms of like, so this is going to be more of a demo and less of slides. Let’s just start. Again, everybody’s talking about Docker Compose at this conference as well, so I have a compose file as well. It defines a Go backend, a database, MySQL, React frontend, and a proxy. I’m not going to go into details because I’m pretty sure you’ve heard a lot about Compose files already. I’m just going to get my application up with Docker Compose.

So it says that all my containers are running. And if I go to my web application, which is now at local osport.tv; it’s running. So it’s a very simple React web app. It allows you to do voting. You can reset it, and you can do voting again. And you can see like just a list of when the last person voted, just the further purpose of testing. So super simple setup with Docker Desktop, moving ahead. So now we connect this application to Grafana. We can do it like so.

Connecting to Grafana

Grafana is a very popular open source software. It has been around as open source software for many years. But generally, people have been having difficulties connecting, getting your data to Grafana. So like we have over the years made it super easy. And this is one of the easiest ways to connect your application to Grafana and start sending metrics and monitor anything. We will just install our Grafana desktop extension. It will start sending metrics to the Grafana Cloud. Let’s see how we can do that.

Grafana Cloud has a very generous free tier. If you want to sign up, you can just go sign up, create a free account. You can log in through Google, GitHub, anything, or just create a new account. Once you have your Grafana Cloud setup, you will be welcomed with an instance like this.

Here, we have a section called infrastructure integrations, which you can just easily access by just typing add new connection here. Our connections page is like a holistic page where you can just connect Grafana to almost any data source in the world, and it provides you many different ways. So for example, if I just start in search connections. I talk about Docker, so we have two integrations. One is for the Docker engine itself. And one is for the Docker Desktop specifically, which allows you to connect this easily.

Docker Desktop integration

Let’s start with the Docker Desktop integration. It says that before installing this Docker Desktop integration, you have to install the Grafana Cloud extension. So if I go to my Docker Desktop, and I just go into the extensions marketplace and browse, or I can use the top search bar and search for Grafana. If you go into extensions, you can see the Grafana Cloud extension there. It’s already installed. I’m just going to open it. Here, you have to go back to Grafana Cloud to basically get your credentials and put them here. I press install, and then it’s asking me either to create a new open API token or like a user existing API token.

Let’s just create a new one. I would say like DockerCon and then create it. And then it has generated an API key. It has given me the connection variables that I need to put in there. So either I can copy/paste them, or we have a very quick shortcut, which just opens it up directly in Grafana Cloud. And you click connect to Grafana Cloud. In a few seconds, it will start sending to Grafana Cloud. In the next step, we have to test the connection. It will let you know that the agent is now sending all the metrics from Docker Desktop to your machine.

View Dashboards

So just to talk a bit about the UI, in the UI, we have a View Dashboards button. We can check the status again. We can check the logs of the Grafana agent, which is basically the container that is sending all your metrics to Cloud. And we can also check our configuration that is used to configure integrations, which I will talk about further.

So I click on View Dashboards. And I can see that I have a number of dashboards, but the most important ones, I’ll just open up Docker Overview, Docker Logs, and Node Overview. This is basically a prebuilt dashboard that’s giving you metrics around the current containers running inside your Docker Desktop machine. So I have six containers, six images. I can see CPU utilization and a lot of other metrics.

Our infrastructure integrations are basically a starting point. It has prebuilt dashboards that are super-cool already, but you can easily edit them, modify them, add anything you want to them. And if I go to the Docker Logs dashboard, I can see that I’m collecting the log files as well from all the containers. I can see the complete log file, see the errors, see the warnings, and everything like that. And then I can also monitor the actual Linux virtual machine, which is actually running inside Docker Desktop. So your Docker engine does not actually run on your machine. It runs inside a Linux virtual machine, which is part of Docker Desktop. These are metrics for that Linux virtual machine.

More integrations

Moving ahead. Now that we installed Docker Desktop, we’ve got two integrations out of the box, Docker and Linux. But, we can also enable more infrastructure integrations to monitor more aspects of our application.

Let’s just see how we can monitor more parts of this application. I will be installing a Go integration, an Nginx integration, and a MySQL integration to actually monitor all these three components individually as well. How can I do that? I just open up a new tab, again, from the top, a new connection, and search for Nginx. Then, it basically gives you the option to at least run the Grafana agent. In our case, our Grafana agent is already running inside our Docker Desktop application, so we can actually skip that. We can come directly to this part here.

This is a prerequisite. Basically, you have to do some configuration for Nginx in order for it to show your logs in this form, then our integration work. But I have already configured that. So I will just go ahead and follow the integration snippet that it shows me to enable the integration. If I write my local hostname here, it gives me this job snippet that says inserting the lines in the environment. But this integration actually works by looking at the log files.

Through my Docker dashboard, I am already collecting the log file of Nginx. So I can directly open up this dashboard. I will just filter my container, look at the proxy container. And I can see that it’s showing me the different metrics for Nginx container. Like total requests are 217K because I’ve been testing it since morning. And different HTTP status 244 requests and some other requests like that. So that’s Nginx. Now we move on to another integration. So now we can basically add a new connection. We will search for the MySQL integration.

And then again, MySQL integration has some prerequisites. You have to create a user or read only user, do some specific configurations. It’s all in the documentation. It’s all in the UI as well. You have to add your instance name, which is generally the hostname of your system. But it can be anything to basically uniquely identify this. So I call it Docker Desktop. Again, I want all the metrics so it gives me a simplified configuration. Now it says that below integrations, you have to add this snippet in order to enable this integration. This configuration is basically my Grafana agents configuration, which is available here.

If I open up the configuration, I can see there is an integration section. And two of three of the integrations are already enabled, CAdvisor agent and NodeExporter. I can now enable the other integration as well. I will just copy a snippet that I already have, which has the appropriate DB user and passwords just to save time. This one. Copy it here. You have to just be careful a bit about the indentations, but everything should be fine. Once I save the configuration, it restarts the Grafana agent with the new configuration that you have put in. And pretty soon, you should start receiving metrics for this integration as well.

One more thing that I forgot is that it’s in the documentation as well, but when you install any extension, it basically installs containers inside Docker Desktop. And those containers usually do not have access to your base machine’s network ports. What you have to do is just execute this command, Docker network connect to bridge, the Docker Desktop extension. It actually connects our agent extension to the network of your machine. Then you can access all the ports of your machine and monitor any metrics that you want.

Let’s go ahead and install our dashboards for MySQL. And let’s go to MySQL because I have enabled the logs integration yet just for the sake of time. Here, I can see that I can see all MySQL metrics as well. It shows me current QDs per second, my thread activity, all these different metrics that are available in my dashboard. So very easily, I have enabled observability for another component of my integration.

One last one. We will go and enable the Go integration. Again, add new connection. I go to Go. Again, for Go, I have some prerequisites on how to enable the GoExporter to start exposing the Prometheus metrics, which I have already done. As you can see, in my application, in the slash metrics part, my Go application is already emitting the metrics in a Prometheus-style format that the integration is expecting. All I need to do is just ask the agent to scrape this endpoint to get all those metrics and send to Grafana Cloud. I’m just going to copy the complete configuration I have here. And pretty soon, now we should also have the metrics for Golang. So I’m just going to skip the test connection part and go directly to my dashboard. Okay, so job should be integrations go, an instance should be Docker Desktop. Okay, so just to line them up, you just see, next, and go runtime metrics.

So I have actually, through this one extension, I have basically now five different dashboards through which I can see all the different aspects of my application. I can see the Linux VM. I can see the Nginx metrics. I can see the MySQL metrics as well as I can see the Go runtime metrics as well.

Going back to our point. Now we have enabled dashboards as well. I haven’t gone into alerts, but alerts are also a very big part of Grafana. Basically, we provide prebuilt alerts with all of our integrations that trigger and send you different alarms if any of your metrics go haywire. And you can also write custom alerts and add new alerts to these integrations as well.

Grafana performance testing

Now that we have done all the monitoring part, we can also go ahead and look at another very useful tool in Grafana Cloud, that is called K6 Performance Testing. We can, you know, just through the CLI, through some simple test cases, load test our application. Okay, so we will install the K6 Testing agent in our machine, we will connect it to Grafana Cloud, and then we will load this application.

Moving back to Grafana Cloud, I open up a new tab again. Just making sure my metrics are coming in. Yes, they are fine. Again, search from the top, you can search for Performance Testing, and then it will open up the K6 Grafana Cloud console. Here, you can create a new project. I can give it a name, web app, DockerCon 2023, and create this project. Once I’ve created this project, you know, it says that you don’t have any tests created yet. So let’s just start testing.

The first tab, which I’ve already done, is to install the K6 agent. So the K6 agent is available on Mac, Linux, Windows, and it gives you all the instructions on how to set it up, then authenticate it with your Cloud instance. Then basically it also lets you know that you can create a simple K6 testing script. It’s a very simple JavaScript syntax that you can use to write all kinds of tests from direct API requests all the way up to browser-based Selenium type tests. Then it says, okay, run a test. So let’s go back to our code repository. Here I do have three different tests. Let’s just open test1.js. Since I’ve created a new project, I just want to make sure that I have the correct project ID inside it. I’ll just copy this project ID, and I have set it up in my environment file, and just put it here.

Web application tests

Now I’m ready to run my web application tests. Let’s just go through this test script first. It basically says that, you know, I have a virtual user. A virtual user is basically one instance of your tester. So 100 virtual users will mean, you know, this application will run with simultaneous 100 requests at your application. You can have a maximum duration —how much time do you want this test to run, which I’ve set to 10 seconds. And then you can, you name your test, and then basically define your test. In my test, I’m just opening the local host app at port 80, and then sleeping for one second. So let’s now run it. Okay, so the command, if you can see it’s a bit down here. It’s “k6 run — out cloud k6-tests/test1.js”. So this means that I want to run this test locally on my machine, and then send all the results back to the cloud.

There is another mode where you can actually run the tests from the cloud as well. But that would require your app to be hosted somewhere public through which the Grafana service can access it, and they will run the test directly from the cloud. Since my app is on my local system right now, I will just test this run locally. I press enter, and then it starts sending requests and testing your app as you have said.

It has given me these outputs in CLI, and it has told me that it was able to generate a thousand requests, and all of them completed successfully without any interruptions. But since we have sent this report to the cloud as well, we can use the Grafana Cloud UI to see them in a much better way. So I go back to my projects. I see this Web App DockerCon 2023. Let me see if it has sent it to the correct project. I have the correct project ID. It should be this; it should be saved… curse of live demos. Anyway, okay, let’s create a new project and set up the case. Let me log into the cloud again. Logged in successfully. And now I have this new project ID. Let me put it directly into the test. And run it again. Fingers crossed, this time it should run and send the results to Cloud. Let’s see.

Now we have this open web app test here. It tells you that it has finished. It has a lot of useful metrics that you can see. It has still told us that it made one key request. How much was the average response time for each request. How many failed and everything like that.

This was a very basic simple test that I used to just open my web app. We can do a more complex test as well. For example, this is a backend stress test, where basically I’m just accessing all the backend APIs directly. So let me just paste my project ID. And you can see that what my test does — it basically calls the ad vote API, it calls the account vote API, and then it calls the get votes API, and then sleeps for one second. Again, I can just run it for a shorter duration. I will run test 2.js. And then I will have the backend stress test report in my browser. I have this here as well.

One last more complex test, which is a Selenium browser-based test, which basically opens up the web app, finds the thumb up icon and clicks on it and tests out. Again, I run it for just a short duration, five seconds, 10 virtual users, and just give it the correct project ID, and then run it here. Okay, it says that the number of iterations can’t be less than your virtual users. Let’s just do two virtual users. That was very quick because only two users; they just opened up the app, clicked on all the buttons, and that’s fine. Go back, and you see your frontend stress test here as well.

Testing applications

Now the most interesting thing about this setup is that we have this testing application. We have Docker Desktop for local app development. We have Grafana for observability. But now we can run it all together. For example, let me tell you what I’m talking about. Let’s go here. Let’s just create a very big test, 1000 virtual units, run it for like 30 seconds or so and that’s just started.

Right now, your CLI is telling you what’s going on with your app, but now — through Grafana Cloud, through the integration of Docker Desktop — you can see all these different metrics. How much CPU is it taking? What’s the performance of your Nginx server as the request increase? How is your system actually behaving based on your frontend test? For example, if I look at my SQL integration, I can see how efficient or inefficient my queries are, how much query per second they are performing, the load on my system. And using these, I can also optimize my app and make sure that they perform, that they behave to the load test correctly. Similarly, you can also monitor like the Go runtime metrics together as well.

Let’s just increase the load a bit and make sure it fails this time. So I’ll just do the backend stress test with like 10,000 virtual users and then for 10 seconds. I just want to make it fail. Just a quick note, this used to fail a lot, but with the last Docker Desktop update, they have actually made some very good CPU optimizations. And it’s taking me a lot of views to actually make it fail. But, yeah, I can see that there are a lot of requests failing. So let’s see how our system is behaving. Let’s go to Nginx. Yeah, I can see like some 500s are popping up. So you can increase the metrics. That’s the limits of your app. Optimize it. Make it work under more pressure. And then, you know, look at all these metrics altogether.

At the start, we talked about having your observability development environment right next to your development environment. This is basically — if I go back to my slides — this whole left section on your site, Docker Desktop, all your applications, is basically your development environment. And all this side on the right, Grafana, Grafana Cloud dashboards, is your observability development environment. So you can use this same environment. You can work on the dashboards. You can add new panels. You can add new metrics in your app and create new panels for them. And basically prepare all this monitoring environment before even going live. All you will need to do when you go live is switch the data sources for these dashboards to actual production metrics. And the production metrics will start coming in. Yeah, so you have like these dashboards. You have these test reports. You have the extension, and you can test it all out.

Moving ahead. Basically this whole setup completes your whole life cycle, which is developed through Docker Desktop. And then observed through Grafana and finally load test through K6. And this is like a whole cycle, where you don’t have to rely on a single person unit test. You can actually do a lot more testing. Make your app a lot more robust. Make it a lot more observable right at the development stage before going live. By using this Grafana extension, you can actually gain access to like 80+ different infrastructure integrations. Docker Desktop and Nginx were like a few of them. If you go to add new connection, you can see we have around 80 different technologies that we integrate with. All of them have simple-to-use instructions that you can use to set them up. And, most of them come with a Docker container that you can directly deploy on your Docker Desktop, start testing with it, start observing it, and trying it out.

Docker Desktop Extensions

Lastly, talking about the whole Docker Desktop extension ecosystem, which is actually one of the main topics here in DockerCon as well. This has actually benefited me personally and us a lot as well. Right now, if you see my Docker Desktop instance, I have these five different extensions installed. I have the Nginx extension installed, through which I can easily configure my Nginx web server, change its configuration and stuff. I have the resource usage one, which actually gives me more graphs to look at. And I have the log explorer and the very useful Docker debug tool, which I actually just tried out after the keynote, which is very awesome. It is actually very awesome. The problem that they were showing, I have faced it like 10, 15 different times. But, yeah, it’s great that they have it now.

Again, building an extension — they have super documentation. When we started out, they were very helpful in guiding us as well. We posted our Grafana Cloud extension, I think around six months ago in the marketplace; it has now over 13,000 downloads. That also tells you the power of having your extension here. And it’s basically free for anyone. We wrote it officially through Grafana Cloud, but anyone can contribute any extension to Docker Desktop, and it will start showing up and help out other users.

Conclusion

To round up what we learned, we learned how to run a web app using the Docker desktop and using Compose. We learned how to install the Grafana Docker Desktop extension and use it to monitor our metrics. We enabled other infrastructure integrations to monitor other parts of our apps. And we also finally saw how we can load test using Grafana K6 and get all the valuable insights right in our local development environment. The code that I was showing you, it is in a GitHub repository. I have put it up in a GitHub repository, which is public, and I will try to share the links.

Some relevant links are Docker Desktop, like Docker has a blog post about this extension as well. There’s a lot of helpful material around the internet on how to use this. And you have Grafana Cloud support as well where you can submit a ticket. Reach out to us or just reach out to me on LinkedIn as well, if something doesn’t work.

As a last note, thanks a lot to the Docker team, Ajeet, Amy, and Benjamin. They helped us a lot in creating this whole Docker desktop extension and testing it out. I think that’s it. Thank you. Hopefully, you all will check out Grafana Cloud, check out the Docker Desktop extension, and use it in your products. Thank you.

Learn more

• Use the Grafana Cloud Docker Extension.
• Unlock Docker Desktop Real-Time Insights with the Grafana Docker Extension
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.
• Subscribe to the Docker Newsletter.