How to utilize OCI Event Notifications to monitor OCI Data Integration (OCI-DI) task runs

In the previous post, we learned about the OCI Data Integration tool and it's core components like Data flows, Pipelines and Tasks etc.

As we saw, OCI-DI data integrations and the pipelines are executed through various Tasks. Now the question arises that what kind of mechanism we can put in place to monitor these run and automatically send notifications to the support team ?

Typically, OCI-DI Task runs result in generating OCI Events when tasks get completed (Success or Fail). Data Integration integrates with OCI public logging and monitoring for visibility by the users.

Let's take a look into the step by step process in setting up notifications on OCI events to monitor and alert on OCI-DI task tuns.

- Navigate to Developer Services and locate Notifications under Application Integration section

- We'll create a Notification topic here. Click Create Topic and let's name it OCI_DI_Notifications

- Now, we'll create a Subscription to this topic. Open the newly created topic and click Create Subscription.

- We can create subscriptions using a variety of protocols:

Email: Associates with an email address as input.
OCI Functions: Associates with a custom Function created in OCI
HTTPS: Associates with a custom HTTPS URL
PagerDuty: Associates with PagerDuty Events URL along with the Integration Key
Slack: Associates with Slack Webhook URL and Token
SMS: Associates with a Cell phone number with its Country Code

- For our use case, we'll use Email Protocol and use the desired Email Address where we want to receive the Alerts/Notifications.

- Once a Subscription is created, we'll see it's in Pending status

- This is because an email will be sent to the recipient and they need to respond to it to confirm that they wish to receive the Notifications from this subscription.

- The email will look like this -

- The recipient clicks Confirm Subscription and they should see this page

- Once the recipient confirms, the subscription will become Active in our OCI Topic.

- At this stage, we need to navigate to Observability and Management, go to OCI Events Service and click Rules.

- We'll be setting up the new OCI Event Rule here.

- Let's click Create Rule. I'll name it OCI_DI Notification Rule

- On the Rule Creation page, we'll see below options:

Condition:

- Event Type: Describes the type of OCI Event
- Attribute: Attribute details of various OCI-DI components such as Applications, Tasks etc.

- Filter Tag: OCI tags to be used as filters, if any.

- We'll use Event Type for our use case.

- Service Name: OCI lists all the available services here. We'll use Data Integration service for our monitoring purpose.

- Event Type: This lists all the possible OCI Events pertaining to a variety of OCI-DI operations at Workspace level and Task level.

- We'll select Task - Begin and Task - End so that we can receive a notification whenever a OCI-DI Task Starts and another one when it ends.

- In the View Example Events page, we can inspect what all information will be included in the Notification in JSON format and we can modify this as per our needs, if desired.

- Click Save Changes to save the Rule.

- At this stage, we've completed the Rule and Subscription for monitoring of OCI-DI Tasks runs. Let's test it.


- Navigate to Data Integration under Analytics and AI

- Navigate to the desired workspace and application.

- Let's locate the desired task and Run it

- The task will execute and complete in some time.

- We should receive two Email alerts, one when the Task run was Started and another one when Task run was Ended.

Notification on Task Start:

Notification on Task End: