File Operations

This section covers how AIOps Studio can be used for perform file based operations

Loadfile - Loading files into AIOps Studio

This section explains how users can upload/load a file (in CSV format) into AIOps Studio. To simplify operations, two example files are provided for users to upload to the local RDA/AIOps environment.

Example CSV files are available from the following links.

1. employees.csv -- This file contains a set of Employee details.

2. hosts.csv -- This file contains a set of IP address/Host details.

Step 1: Download the above two files to your local file system using a standard web browser.

Downloaded files under RDA-->file-operations directory

Step 2: Upload (load) the above two into your RDA system as shown below.

• Using a web browser, connect to the RDA system <https://<rda-ipaddress>:9998>

• Click on the CFXDX Python 3 icon from the main landing page of the RDA launcher (as shown below)

• Launch AIOps studio using the RDA provided text field to start AIOps studio.

Type "studio()" into the text field of RDA to load AIOps studio UI

• Upload the two files using the file browser upload button provided as part of RDA UI as shown below:

• Your browser will pop up a file browser window (based on your working environment) to select the downloaded files employees.csv and hosts.csv as shown below.

• Select both the files and upload both the files into your RDA environment. After you have completed uploading files, you will see two files under the file browser of the RDA environment as shown below.

Once a user uploads the files, the RDA file browser will save the uploaded files.

• Now you are ready to load the above two files into AIOps Studio as explained in the following steps.

Step 3: Add a new empty pipeline with the name "load-employee-file" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).

Step 4: Add the following pipeline commands into the empty pipeline text field that you have created in the above Step 3.

You can copy the below code into your pipeline and execute that in your environment. ##### This pipeline loads employees.csv file into AIOps Studio. ##### AIOps studio stores the data loaded from employees.csv file ##### into local dataset named 'employee-dataset'. ##### prints the data that was stored @files:loadfile filename = "employees.csv" --> @dm:save name = ' employee-dataset' --> *dm:filter *

Step 5: Click the button 'Verify' and Click the 'Save' button (AIOps studio verifies the pipeline syntax and stores the pipeline into the earlier created load-employee-file pipeline as shown below.

AIOps studio verfies the syntax of the user created pipline and prints "OK" per step.

Step 6: Users can now execute the above-created pipeline by clicking the 'Execute' button. Once the users click the 'Execute' button, AIOPs studio executes the pipeline that will load the user-defined file, save the loaded file into a local dataset named 'employee-dataset' as shown below.

Step 7: Verify the data using AIOps Studio using 'Inspect --> Data --> Get Data'.

Get Data button will provide the data from the dataset that was loaded

employee.csv data stored and displayed by AIOps Studio using simple pipeline.

Step 8: Try the above steps with the hosts.csv file to store/display the data loaded by AIOps studio using file load operation.

##### This pipeline loads 'hosts.csv' file int AIOps Studio ##### AIOps studio stores the data loaded from hosts.csv file ##### into local dataset named 'hosts-dataset'. ##### prints the data that was stored @files:loadfile filename = "hosts.csv" --> @dm:save name = ' hosts-dataset' --> *dm:filter *

Savefile - Saving data/dataset output to a file from AIOps Studio

This section explains how users can save the data/dataset output from a pipeline into a file (in CSV format) via AIOps Studio. To simplify operations, two examples are provided for users to save to the local filesystem using AIOps studio.

Example CSV files are available from the following links.

1. car-model-to-price.csv - This file contains a file that was saved from the data output of pipeline execution saved to a file.

2. device_details.csv -- This file contains a set of device details saved from the data output of pipeline execution saved as a file.

Step 1: Add a new empty pipeline with the name "car-model-pipeline" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).

Step 2: Add the following car-model-to-price pipeline as shown below using the AIOps pipeline.

You can copy the below lines of code into your pipeline and execute that in your environment.

##### This pipeline creates temporary car models to price dataset ##### and saves the temporarily created dataset into car-models.csv ##### File will be saved under ~/cfx/cfxdx/home/nodebooks directory. @dm:empty --> @dm:addrow car_model = 'Honda Civic' & price ='16000' & id = 'hc1' --> @dm:addrow car_model = 'Toyota Corolla' & price ='13000' & id = 'tc2' --> @dm:addrow car_model = 'Ford Focus' & price ='11000' & id = 'ff3' --> @dm:addrow car_model = 'Audi Q5' & price ='49000' & id = 'aq5' --> @files:savefile filename = 'car-model-to-price.csv'

Once the above code is added to the empty pipeline that was created via AIOps studio, click the 'Verify' button to verify the pipeline syntax is valid using the studio.

PIpeline details for car-models-to-price-pipeline.

Step 3: Execute the pipeline from AIOps studio. AIOps studio executes the pipeline and prints the output as shown below.

Execution sequence and also saving the data to filename is displayed above.

The file saved to notebooks directory and file path as shown.

Step 4: Execute another pipeline from AIOps studio using the following. AIOps studio executes the pipeline and prints the output as shown below.

##### This pipeline creates network device details dataset ##### and saves the temporarily created dataset into device_details.csv ##### File will be saved under ~/cfx/cfxdx/home/nodebooks directory. @dm:empty --> @dm:addrow fqdn = 'itm1.cfx.com' & ip_addr ='192.168.1.1' & id = '1' --> @dm:addrow fqdn = 'dev1.cfx.com' & ip_addr ='192.168.10.1' & id = '2' --> @dm:addrow fqdn = 'qa1.cfx.com' & ip_addr ='192.168.2.1' & id = '3' --> @dm:addrow fqdn = 'fw.cfx.com' & ip_addr ='10.95.101.1' & id = '4' --> @files:savefile filename = 'device_details.csv'

AIOps Studio saving the dataset output of a pipeline to a CSV file (device_details.csv)

Appendfile - Save to file using filename, if a file already exists, append the data.

Note: Supports only CSV format.

This section explains how users can append additional data from the dataset output generated from a user-defined pipeline into a file (in CSV format) via AIOps Studio. To simplify operations, examples are provided for using the existing example files.

Example CSV files are available from the following links.

1. append-to-device-details.csv -- In this example, we will use a file (append-to-device-details.csv) from the file system that has a set of devices already added and has other related values (e.g. network device details, etc). After, execution of the pipeline, additional/new data is added/appended to the existing file.

append-to-device-details.csv file stored on the file system

Content of the append-to-device-details.csv before executing the pipeline

Step 1: Add a new empty pipeline with the name "append-to-device-details-pipeline" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio)

Add a new pipeline with name 'append-to-device-details-pipeline' and save.

Step 2: Add the following car-model-to-price pipeline as shown below using the AIOps pipeline.

You can copy the below lines of code into your pipeline and execute that in your environment.

##### This pipeline creates network device details dataset ##### and saves the temporarily created dataset ##### into append-to-device-details.csv ##### File will be saved under ~/cfx/cfxdx/home/nodebooks directory. @dm:empty --> @dm:addrow fqdn ='dev-ops1.cfx.com' & ip_addr='192.168.1.10' & id = '5' --> @dm:addrow fqdn ='ops1.cfx.com' & ip_addr='192.168.10.14' & id = '6' --> @dm:addrow fqdn ='router11.cfx.com' & ip_addr='10.95.101.12' & id = '7' --> @files:appendfile filename = 'append-to-device-details.csv'

Step 2: Verify the above lines of code via AIOps studio using the 'Verify' button on the UI as shown below. AIOps studio will verify the syntax of the pipeline code and display 'OK' to proceed to the next step.

Once the user adds the pipeline code, click 'Verify' button. AIOps studio verifies and displays 'OK' for all the pipeine steps

Step 3: Next step is to execute the pipeline code via AIOps studio by selecting/clicking the 'Execute' button as shown below. Once the studio, executes the pipeline, it will display the successful execution of steps on the screen. In addition, the existing file will have additional data/dataset appended to the existing file as shown in the below screen capture.

Step 4: Verify the data is appended to the existing data using your favorite editor as shown below.

AIOps studio pipeline

Datasets to excel file - Export specified datasets to an Excel file, each dataset as a different sheet. Returns summary of export.

This section explains how users can create an excel file with multiple tabs having different datasets via AIOps studio.

Example excel file is available from the following link.

1. router-switch-firewall.xlsx : This is an example excel file that is generated after executing the example pipeline.

Step 1: Add a new empty pipeline with the name "datasets-to-xlsx-pipeline" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).

Create an empty pipeline for adding pipeline code

Step 2: Add the following datasets-to-xlsx-pipeline code as shown below using the AIOps pipeline.

You can copy the below lines of code into your pipeline and execute that in your environment.

##### This pipeline creates multiple logical datasets that in turn will be used ##### to save to an excel file with multiple tabs (for each dataset) ##### into router-switch-firewall.xlsx ##### File will be saved under ~/cfx/cfxdx/home/notebooks directory ##### Router block of elements -- block-1 --> @c:new-block --> @dm:empty --> @dm:addrow fqdn ='router1.cfx.com' & ip_addr='192.168.1.1' & id = '1' --> @dm:addrow fqdn ='router2.cfx.com' & ip_addr='192.168.10.1' & id = '2' --> @dm:addrow fqdn ='router3.cfx.com' & ip_addr='192.168.11.1' & id = '3' --> @dm:addrow fqdn ='router4.cfx.com' & ip_addr='192.168.121.1' & id = '4' --> @dm:save name = 'router-dataset' ##### Switch block of elements -- block-2 --> @c:new-block --> @dm:empty --> @dm:addrow fqdn ='switch1.cfx.com' & ip_addr='192.160.1.1' & id = '1' --> @dm:addrow fqdn ='switch2.cfx.com' & ip_addr='192.161.10.1' & id = '2' --> @dm:addrow fqdn ='switch3.cfx.com' & ip_addr='192.162.10.1' & id = '3' --> @dm:addrow fqdn ='switch3.cfx.com' & ip_addr='192.163.10.1' & id = '4' --> @dm:save name = 'switch-dataset' ##### Firewall block of elements -- block-3 --> @c:new-block --> @dm:empty --> @dm:addrow fqdn ='firewall1.cfx.com' & ip_addr='192.170.1.1' & id = '1' --> @dm:addrow fqdn ='firewall2.cfx.com' & ip_addr='192.171.1.1' & id = '2' --> @dm:addrow fqdn ='firewall3.cfx.com' & ip_addr='192.172.1.1' & id = '3' --> @dm:addrow fqdn ='firewall4.cfx.com' & ip_addr='192.173.1.1' & id = '4' --> @dm:save name = 'firewall-dataset' ##### This block is adding an empty dataset first, followed by ##### three logical datasets that were created from earlier above ##### pipeline blocks -- block-4 --> @c:new-block --> @dm:empty --> @dm:addrow dataset = "router-dataset" & sheet = "Router" --> @dm:addrow dataset = "switch-dataset" & sheet = "Switch" --> @dm:addrow dataset = "firewall-dataset" & sheet = "Firewall" ###### pipeline block which stores the logical datasets into final output file -- block-5 --> @c:new-block --> @files:datasets_to_xlsx filename = "router-switch-firewall.xlsx" & datasets = "router-dataset|switch-dataset|firewall-dataset"

The above set of pipeline code blocks are explained below for each step. The following explanation will clarify mu

Router block of elements -- block-1: In this pipeline block, we are trying to create a simple dataset of three rows for 'router' related data (example data per se). @c:new-block: In the first line, '@c:new-block' is a keyword to tell studio runtime that the user wants to start a logical new block. @dm:empty: In the second line, '@dm: empty' is a keyword (data mapper functionality) to tell the studio runtime that the user wants to start an empty dataset to start with. @dm:addrow: The rest of the lines have '@dm:addrow' tells the studio to add data for each new row into the dataset using different values as shown in the above pipeline code.

Step 3: Add the above pipeline code into the newly created pipeline in Step-2, verify the pipeline using the 'Verify' button. The studio will verify the syntax and display 'OK' as shown below.

Code added to the pipeline in Studio.

Studio verifies all the blocks and prints 'OK'

Step 4: Once the pipeline code syntax is verified, run the pipeline code by selecting the 'Execute' button. Studio will execute the pipeline as shown below and stores the dataset(s) into excel file under ~/cfx/cfxdx/home/notebooks folder. Output screenshot is shown below for reference:

First 11 steps (after execution)

Steps 11 - 19 (sfter execution)

Steps 19 - 28 (after completion of the pipeline)

Step 5: Now, users will be able to verify the data written into an excel file using the windows excel tool.

Note: You can cross-check the generated data using the reference file 'router-switch-firewall.xlsx'

Output file generated at ~/cfx/cfxdx/home/notebooks.

Dataset to JSON file - Section explains how to store dataset into a file in JSON format.

This section explains how users can create a dataset using CSV file via AIOps studio. In turn, use AIOps RDA functions to store the imported data from CSV file into the dataset to

Download the incidents.csv file to the local machine as shown below using a standard web browser.

Step 1: Download 'incidents.csv' to AIOps RDA environment as shown below from the local file system.

Downloaded file on local filesystem

Step 2: Upload the file 'incidents.csv' to AIOps studio using file-browser (as shown below)

Screenshot displays how to upload a file into AIOps Studio.

Step 3: Add a new empty pipeline with the name "load-incidents-file" as shown below and click the "Save" button (this step will create an empty pipeline and saves it to AIOps studio).

Create an empty pipeline.

Step 4: Add the following pipeline commands into the empty pipeline text field that you have created in above Step 3.

You can copy the below code into your pipeline and execute that in your environment. ##### This pipeline loads incidents.csv file into AIOps Studio. ##### AIOps studio stores the data loaded from incidents.csv file ##### into local dataset named 'incident-summary'. ##### prints the data that was stored @files:loadfile filename = "incidents.csv" --> @dm:save name = 'incidents-summary' --> *dm:filter *

Step 5: Click the button 'Verify' and Click the 'Save' button (AIOps studio verifies the pipeline syntax and stores the pipeline into the earlier created load-incidents-file pipeline as shown below.

Step 6: Users can now execute the above-created pipeline by clicking the 'Execute' button. Once the users click the 'Execute' button, AIOPs studio executes the pipeline that will load the user-defined file, save the loaded file into a local dataset named 'incident-summary' as shown below.

Creating dataset that will be used to convert to JSON format from CSV

Step 7: Add the following code snipped to the above created pipeline

You can copy the below code into your pipeline, verify and execute that in your environment. ##### This pipeline loads incidents.csv file into AIOps Studio. ##### AIOps studio stores the data loaded from incidents.csv file ##### into local dataset named 'incident-summary'. ##### prints the data that was stored @files:loadfile filename = "incidents.csv" --> @dm:save name = 'incidents-summary' --> *dm:filter * ###### This block is adding an empty dataset first, followed ###### by restoring the dataset that was created above and next, ###### saving the data into 'incidents.json' --> @c:new-block --> @dm:recall name = 'incidents-summary' --> @files:savefile filename = 'incidents.json' -

Step 8: User can now execute the pipeline using AIOps Studio. Once the pipeline is executed, RDA loads the dataset and stores the data into 'incidents.json' (in JSON format) as shown below.

File operations next steps

Now, users will able to use the above reference examples to try with other CSV/data files.