Manipulating Data from Oracle Object Storage to ADW

Posted 2020-11-24 ak666666

 

0. Introduction and Prerequisites

This article presents an overview on how to use Oracle Data Integrator in order to manipulate data from Oracle Cloud Infrastructure Object Storage on OCI. The scenarios here present loading the data in an object storage in Oracle Cloud Infrastructure and then move the data to ADW.

This could be a reference for customer have data storage in different regions and want to do the integration& feed into Data warehouse.

You should have Object storage and ADW instance provisioned.

1. Install ODI 12.2.1.3.0

Not include in this document.

2. Patch p26669648_122130_Generic to upgrade to 12.2.1.3.1

You need to patch ODI to version 12.2.1.3.1 firstly.

3. Now you get ODI 12.2.1.3.1

4. New a Data Server

Let’s setup the topo. Right click Oracle Object Storage

An overview:

a. Region:

Oracle Object Storage region. A region is a localized geographic area, and an availability domain is one or more data centers located within a region. A region is composed of several availability domains. Most Oracle Cloud Infrastructure resources are either region-specific, such as a virtual cloud network, or availability domain-specific, such as a compute instance.

b. Tenant OCID:

Tenant’s Oracle Cloud ID. Every Oracle Cloud Infrastructure resource has an Oracle-assigned unique ID called an Oracle Cloud Identifier (OCID). It‘s included as part of the resource‘s information in both the Console and API. To find your tenancy‘s OCID.

Tennent OCID:

ocid1.tenancy.oc1..aaaaaaaamsltvh6ebmb3l7gw7nu424jbp2dkdszlrejmy7huthni4pakazfa

c. User OCID:

Oracle Cloud ID of the user logging into Oracle Object Storage.

In the Console on the page showing the user‘s details. To get to that page:

• If you‘re signed in as the user, click the user icon present in the top-right corner of the Console, and then click User Settings.
• If you‘re an administrator doing this for another user, instead click Identity, click Users, and then select the user from the list.

User OCID: api.user

ocid1.user.oc1..aaaaaaaast2jhktt5tfgljrmrqer7osauny6r2l6vgvdnx2bukopaqfyu3na

Security

d. Private Key File – Click the browse button to choose the location of the private key file (in PEM format)

Follow the steps to generate the private key and fingerprint

https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm#How

·       Passphrase – Passphrase is the password used while generating the private key

Password: oracle

e. fingerprint

f. username:

Specify the user api.user, need to be same with item c. User OCID

Caution: Upload the public key to Object Storage.

You can upload the PEM public key in the Console, located at https://console.us-ashburn-1.oraclecloud.com. If you don‘t have a login and password for the Console, contact an administrator.

1. Open the Console, and sign in.

2. View the details for the user who will be calling the API with the key pair:

• If you‘re signed in as this user, click your username in the top-right corner of the Console, and then click User Settings.

• If you‘re an administrator doing this for another user, instead click Identity, click Users, and then select the user from the list.

• Click Add Public Key.
• Paste the contents of the PEM public key in the dialog box and click Add.

Test Connection

5. Creating an Oracle Object Storage Physical Schema

 

Create an Oracle Object Storage physical schema using the standard procedure, in Administering Oracle Data Integrator.

Oracle Object Storage specific parameters are:

·       Name: Name of the physical schema created

·       Bucket (Schema): It specifies the Oracle Object Storage Bucket name from which upload, download or the delete operation will happen. Select the required bucket from the Bucket Name drop-down list.

·       Directory (Work Schema): This is the temporary folder on the local system used for getting files from Oracle Object Storage bucket during reverse engineering. If the directory does not exist it will be created. Specify the required location in the local system.

And the logical schema

 

6. Creating and Reverse-Engineering an Oracle Object Storage Model

Creating an Oracle Object Storage Model

An Oracle Object Storage model is a set of data stores, corresponding to files stored in an Oracle Object Storage bucket. In a given context, the logical schema corresponds to one physical schema. You can create a model from the logical schema for the Oracle Object Storage technology. The bucket schema of this physical schema is the Oracle Object Storage bucket containing all the files. You can create new ODI Data store that will represent a file in Oracle Object Storage so that it can be used in mappings.

Input the information required and Save.

Reverse-Engineering Delimited Files from Oracle Object Storage

To perform a delimited file reverse engineering:

1.       In the Models accordion, right click your Object Storage Model and select New Data store. The Data Store Editor opens.

2.       In the Definition tab, enter the following fields:

·       Name: Name of this data store

·       Resource Name: Click the Search icon, to select the required file from the list of files present in Oracle Object Storage for the configured bucket.

3.       Go to the Storage tab, to describe the type of file. Set the fields as follows:

·       File Format: Delimited

·       Heading (Number of Lines): Enter the number of lines of the header. Note that if there is a header, Oracle Data Integrator uses the first line of the header to name the columns in the file.

·       Select a Record Separator.

·       Select or enter the character used as a Field Separator.

·       Enter a Text Delimiter if your file uses one.

·       Enter a Decimal Separator, if your file contains decimals.

4.       From the File main menu, select Save.

5.       In the Data Store Editor, go to the Attributes tab.

6.       In the editor toolbar, click Reverse Engineer.

Click Reverse Engineer, ODI will generate the Metadata based on the header of the file.

7.       Verify the data type and length for the reverse engineered attributes. Oracle Data Integrator infers the field data types and lengths from the file content, but may set default values (for example 50 for the strings field length) or incorrect data types in this process.

8.       From the File main menu, select Save.

7. Create a Connection with ADW

Create a Data Server for ADW.

JDBC information will be there, no need to update.

And Test the connection

And new a Physical Schema.

New the Model and Reverse Engineer.

8. New a Project, Mapping and Test

 

 

Set the AP as below:

 

Caution: You need to run create credential store procedure on ADW before running the Mapping

set define off

begin

DBMS_CLOUD.create_credential(

credential_name => ‘ODI‘,

username => ‘AK‘,

password => ‘.};rKwO6t8pKzl2mp7U3‘

);

end;

/

set define on

 

Mapping run finished.

And Review data in ADW.

Comparing with the source csv file:

Generate Scenarios: