While Google Cloud offers many preconfigured metrics to monitor various aspects of your services, there will be times that you need more granularity or a focus on specific events. In such instances, Google Cloud provides the flexibility to define custom metrics. These can be broadly categorized into two types: log-based metrics and user-defined metrics.
Log-based metrics are derived directly from the logs your application generates. They allow you to quantify and monitor particular events or patterns that appear in your logs. For instance, as you just saw, each time the skill service initializes, it logs the number of tags loaded from the file stored in Google Cloud Storage. This action is logged, noting the exact count of tags loaded. With log-based metrics, you can create a custom metric that counts the number of times the tags are retrieved. You can also create another that monitors the number of tags loading, allowing you to notice if this number changes. If this number is significantly lower than the average, it is likely there is an issue that needs to be investigated.
Creating a counter metric from logs
As you just saw, there is a log entry created each time the skill service starts, logging the number of tags loaded from Cloud Storage. Similarly, there is a log entry created each time a request for a tag suggestion has been processed. Here is an example:
autocomplete for java took 84.714804ms
A custom counter metric will enable you to record the number of suggestions by all instances of the skill service. You cannot create this type of metric with a gcloud command alone; you need to define it in a file and then use the file to create the metric.
You have now created a user-defined, log-based counter metric. However, you will need to add it to a dashboard to see it. Before that, create another type of metric to show with it.
Creating a distribution metric from logs
It would also be useful to extract the time it took to make the suggestion from the log entry itself. This is possible with a distribution metric.
Instead of just counting the number of logs, you can extract the value from the log entry with a regular expression. Again, you will need to define it in a file and then use the file to create the metric with gcloud.
These metrics will then be available to add to a dashboard, and there will be some data to display.
Creating a dashboard with user-defined metrics
To add the metrics to a dashboard, you will need to create a configuration file. The file templates/tag-dashboard.yaml.template is a template for the dashboard configuration file. It contains placeholders for the project ID and service name.
You will see a new dashboard named “skills-service performance.” Open this and you will see two charts, one visualizing the number of suggestions made per minute and the other showing the time taken to respond to the 50% percentile of requests.
