TIBCO Spotfire: Creating Custom Narratives – JSON-Type Studio Projects​


Documentation for Arria NLG’s Add-in for TIBCO Spotfire Analyst


 

This document is about BI integration with JSON-type Studio projects. If you are working with tabular data – that is, spreadsheet data – see Creating Custom Narratives – Table-Type Studio Projects. It is organized as follows:

Creating a custom narrative

Add Arria’s custom visual to a sample Spotfire analysis
Configure a narrative to describe the visuals
Create a project in Arria NLG Studio
Modify the Studio project
Publish the Studio project
Link your published Studio project to your Spotfire analysis

Using mapping scripts

Optimizing the narrative to handle filters

Contextualizing the narrative

Going further

Other documentation for Arria’s add-in for TIBCO Spotfire Analyst

 

 

Creating a Custom Narrative

This guide explains how to customize a starter project using Arria NLG’s add-in for TIBCO Spotfire and Arria NLG Studio. You’ll begin by downloading a sample Spotfire analysis. Next, you’ll import Arria’s custom visualization into the report. After that, using Arria’s custom visualization, you’ll add a narrative wizard to the report, in which you’ll provide some information about your data and the kind of things you would like to say about it.

Then, using NLG Studio, you’ll customize a very simple NLG JSON-type project, which you’ll publish to the cloud. By moving through the steps of customizing an NLG project, you’ll get a glimpse of the amazing things you can do when you extend Spotfire with Arria’s NLG technology.

Finally, you’ll link the published Studio project to the narrative wizard to bring its NLG into the report.

Prerequisites

 

 

Add Arria’s custom visual to a sample Spotfire analysis

1. Download the Customizing Narratives sample Spotfire analysis (.DXP file).

2. In Spotfire, open the sample analysis.

3. In the Authoring bar, click Visualizations > Add visualization….

4. Select ArriaNLG at the bottom of the Visualization types… list.

5. Move and resize the Arria add-in to suit your screen.

 

Configure a narrative to describe the visuals

1. In the ArriaNLG add-in, click on the sliders icon to begin selecting data for your narrative.

2. From Dimensions, select SegmentProduct and Year.

3. From Measures, select Sales (not Sale Price or Gross Sales), Profit.

Now that you’ve selected the data you wish to describe, you’re ready to bring that subset of data into NLG Studio.

4. Click the Create Custom Narrative tab.

5. Make sure the Simple/Advanced toggle is set to Simple.

6. Make sure the Studio Project Type is set to Describe a JSON Object. (This is because the NLG Studio starter project you are going to work with is a JSON-type project. If the Studio project you associate with your dashboard is based on CSV data, you would choose Describe a Table here.)

7. Click on View Dataset in JSON to view your data selection JSON format.

8. Click the download icon to download the data.

9. Check your machine’s Downloads folder for the file called data.json. In the next section, you’ll import this into the BI Starter project in NLG Studio.

 

Create a project in Arria NLG Studio

You are going to create a very simple sample project in NLG Studio in order to get a basic understanding of how you can use NLG Studio to augment your analytics. (For instructions that show how to create a more elaborate narrative, see the NLG Studio documentation.)

1. Log into NLG Studio.

2. In the Project Portal, click on NEW PROJECT and select New Project from Sample > BI Starter.

3. Rename the project if required.

Note: This project may become the starting point for all BI projects you create in NLG Studio in future. You may wish to rename the project to something generic, or to give it a name that matches the name of the corresponding .DXP analysis file. To change the name of the project, just click in the name field and start typing.

4. Click the sample project to open it.

You’ll see the data that comes with the starter project. When you’re ready to modify this starter project, you’re going to replace this data with the data from your field selections in Spotfire.

5. In the left navigation ribbon, click Compose to go to the Compose view.

In the editor, you’ll see the contents of the Main script, which is empty aside from a reference to a ReadMe sub-script. This ReadMe describes the BI data format used in Spotfire projects, and provides information about the helper functions in the project that appear as a list of sub-scripts under the Main script. These helper functions are user-defined functions created specifically for working with BI data.

6. Make sure you’re in the Main script, then at the top right of the screen, click Preview. You’ll see a preview of the information contained in the ReadMe in the Output Text section.

In the next section, you’ll make some changes to this starter project.

 

Modify the Studio project

You are going to compose a piece of narrative describing the total sales and total profit for the selected period. To generate the text for that, you’ll add some code in the Compose editor. But first, you need to import the data set that is specific to the fields you have selected in MicroStrategy: Product, Segment, Year, Profit, Sales.

1. In the left navigation ribbon, click Data to go to the Data view.

2. At the top right of the screen, click the import icon to bring in the file called data.json from your Downloads folder.

3. Click Compose to go to the Compose view.

4. Click Main to begin editing the Main script.

Note: When working through these tutorials, it is best to activate Plain Paste Mode before editing scripts in NLG Studio. This ensures that any HTML formatting (such as text color) is removed from copied text before you paste it. Click on the the clipboard icon in the editor ribbon to toggle between Formatted Paste Mode and Plain Paste Mode.

Studio remembers the last paste mode you selected.

5. Remove the text that is already there ([[ReadMe]]).

6. Copy/paste the following:

For the [[inflectNoun('year',len(getUniqueValues(data,'Year')))]] [[getUniqueValues(data,'Year')]], the total sales were [[formatCurrency(columnAggregate(data,'Sales','sum'),'USD','#.00','')]] and the total profit was [[formatCurrency(abs(columnAggregate(data,'Profit','sum')),'USD','#.00','')]].

7. In the top right of the screen, click Preview. Your previewed output should look like this:

Now add another piece of narrative. You’ll add a sentence reporting on the sectors that contributed the most and least profit. As an exercise, you’ll create some in-script variables that you’ll use to replace some of your existing code.

8. In Main, replace the existing script with the following:

[[global.currency='USD';
years=getUniqueValues(data,'Year');
totalSales=columnAggregate(data,'Sales','sum');
totalProfitLoss=columnAggregate(data,'Profit','sum');
profitLossbySegments=groupBy(data,'Segment','Profit','sum');
topSegmentbyProfit=top(orderDesc(profitLossbySegments),1)[0];

if (currency !=null){For the [[inflectNoun('year',len(years))]] [[years]], the total sales were [[formatCurrency(totalSales,currency,'#.00','')]] and the total [[direction(totalProfitLoss ,'profit','loss' )]] was [[formatCurrency(abs(totalProfitLoss),currency,'#.00','')]].}]]

 

9. Click Preview. Your previewed output should look like this:

10. Now add another sentence to your narrative, this time reporting on the loss made in some segments. Compose this sentence in a new script.

11. At the top of the list of scripts, click Add Script and overwrite the NewScript name with the name describeContrast.

12. In this new sub-script, copy/paste the following:

#define describeContrast(category,currency)[[salesForSegment=columnAggregate(filterOn(data,'Segment','==',category[0]),'Sales','sum');

if (category[1]<0){Even though the [[category[0]]] segment posted [[formatCurrency(salesForSegment,currency,'0.##','')]] sales, it reported a loss of [[formatCurrency(abs(category[1]),currency,'0.##','')]].}]]

In order to call the new sub-script, you need a new variable that represents the segment with the lowest profit (or highest loss). You can create an in-script variable in Main to do this.

13. Go back to the Main script and overwrite what you have there with the following code (which contains the new variable bottomSegmentbyProfit and the call to the new sub-script describeContrast):

[[global.currency='USD';
years=getUniqueValues(data,'Year');
totalSales=columnAggregate(data,'Sales','sum');
totalProfitLoss=columnAggregate(data,'Profit','sum');
profitLossbySegments=groupBy(data,'Segment','Profit','sum');
topSegmentbyProfit=top(orderDesc(profitLossbySegments),1)[0];
bottomSegmentByProfit=top(orderAsc(profitLossbySegments),1)[0];

if (currency !=null){For the [[inflectNoun('year',len(years))]] [[years]], the total sales were [[formatCurrency(totalSales,currency,'#.00','')]] and the total [[direction(totalProfitLoss ,'profit','loss' )]] was [[formatCurrency(abs(totalProfitLoss),currency,'#.00','')]]. The [[printKey(topSegmentbyProfit)]] segment reported the most [[direction(topSegmentbyProfit[1], 'profit','loss')]]: [[formatCurrency(abs(printValue(topSegmentbyProfit)),currency,'0.##','')]]. [[describeContrast(bottomSegmentByProfit,currency)]]}]]

14. Click Preview. You should see the following:

Publish the NLG Studio project

1. In your NLG Studio project, click Publish in the left navigation ribbon to go to the Publish view.

2. In the Publish > API tab, click the Publish button.

3. Make a note of the project URL. (From inside the URL box, you can click the copy icon to copy the URL to your clipboard. Paste it to a notepad.)

4. In the API Key section, click the Generate API Key button.

5. Make note of the API key generated. (Again, you can copy it to your clipboard from inside the box with the key. Paste it to a notepad.)

Note: The NLG Service URL and API key provided here are only examples; they will not work in practice. You must use your own Studio project URL and API key.

 

 

Link your published NLG Studio project to your Spotfire Analysis

Now you’re ready to configure the Arria add-in to use your newly published Studio project.

1. Back in Spotfire, in the first empty field in the Arria add-in, enter the published Studio project URL you made a note of earlier.

2. In the second empty field, paste in your API key.

3. Click the Generate Text button. You should now see your customized narrative.

4. Click around on different pieces of the charts and use the filters to see how the narrative responds. To select multiple bars at the same time, use CTRL+Click.

5. (Optional) You can copy the narrative for pasting elsewhere (e.g. into Microsoft PowerPoint or Word) by clicking the copy icon in the top right of the widget.

 

 

Using Mapping Scripts

With Arria’s add-in, you can add custom JavaScript to modify the input data set — right inside the narrative widget. You do this in the Edit Mapping Script section.

About the Methods Used:

  • getData(): This method returns the underlying JSON data (the data you see in the View Dataset in JSON section).
  • addMetaData(appSpecificData,key,value);: This method appends a key-value pair into the metadata object of the dataset (appSpecificData).
  • setData(appSpecificData): This method sets the appSpecificData variable (the transformed data object). After using this method, the transformed data object is available for download and posting to the NLG service URL.

The following example shows how you can assign some variable values (var data=getData() ;) to the input table data object that can be used in the narrative script:

var data=getData();
addMetaData(data,"currency","GBP");
setData(data);

Note: In the above script, we add the "currency" : "GBP" key-value pair as metadata. 
However, with
addMetaData, you can add any key-value pairs that you want to pass to the Studio project, for example a threshold as shown here:

var data=getData();
addMetaData(data,”currency”,”GBP”);
addMetaData(data,”threshold”,5.3);
setData(data);


To try out a custom mapping script:

1. Click on the sliders icon to take you back to the narrative add-in settings.

2. Copy the above script and paste it in the Edit Mapping Script section.

3. In the View Dataset in JSON section, at the top right, click the download icon to download the data. This time you’ll get the sample data again, but with your customization.

If you examine the downloaded data, you’ll see the additional information: “metadata”:{“currency”:“GBP”}

4. Click Open with Studio to open your BI Starter project.

5. Now assign the currency metadata to a variable called currency. You can do this easily as an in-script variable.

Replace the first line of the current Main script with the following: [[global.currency=WholeJSON['dataset']['0']['metadata']['currency'];

 

To see the results:

1. In order to preview, you first need to import the modified sample data into the Studio project. (This is the data you modified using the custom mapping JavaScript.)

a. In the left navigation ribbon, click Data to go to the Data view.
b. At the top right of the screen, click the Import new sample data icon.
c. Browse, or drag and drop, to import the modified data file from your Downloads folder.
d. Click Continue.

In the Data view, you should see the new "currency": "GBP" metadata.

2. Go to the Compose view, then from the Main script, click Preview to see the currencies expressed in sterling (£) instead of dollars (US$).

3. Now re-publish the Studio project. The URL endpoint and the API key remain the same, so you don’t need to update anything more in the narrative configuration window in your Spotfire analysis.

4. Go back to Spotfire. Refresh the narrative (click Generate Text in the Arria add-in).

You should see the currencies expressed in sterling (£) instead of dollars (US$).

Optimizing the Narrative to Handle Filters

At this point, in your Spotfire analysis, if you click on a single sector within a graph, the narrative still describes the sector with the most profit or loss. But this information is redundant because the selected data represents only a single sector.

To see an example of this redundancy, try this:

1. In the top graph, select only the Government segment.


2. Now check out how the narrative changes.

Do you see the second sentence? It’s reporting on the greatest profit by segment, which happens to be the Government segment. But because we’ve selected only the Government segment, we’re already getting the profit figure for that segment in the first sentence. We don’t need the second sentence. The next exercise will show you how to filter that out.

NOTE: By removing that second sentence, we are removing only one redundancy. The narrative could be further optimized. But we’ll leave that for another time.

So, focusing on one fix here, you can filter out that redundant sentence — the sentence describing the segment with the most profit or loss — when there is only one segment selected.

To filter out the sentence when it is irrelevant:

1. In NLG Studio, in your BI Starter project, locate the following sentence in your Main script:

The [[printKey(topSegmentbyProfit)]] segment reported the most [[direction(topSegmentbyProfit[1], 'profit','loss')]] of [[formatCurrency(printValue(topSegmentbyProfit),currency,'0.##')]].  

2. Add the following if statement surrounding the sentence (include the curly brackets and replace <above script> with the sentence above):

[[if (len(profitLossbySegments)>1){ <above script> }]]

i.e., replace the sentence with this one:

[[if (len(profitLossbySegments)>1) { The [[printKey(topSegmentbyProfit)]] segment reported the most [[direction(topSegmentbyProfit[1], 'profit','loss')]]: [[formatCurrency(abs(printValue(topSegmentbyProfit)),currency,'0.##','')]]. }]]

Note: If you preview the narrative at this point, you’ll notice that the sentence still appears in the narrative within NLG Studio. This is because Studio is working with the data it currently has, not the data that comes from the selection of only the Government segment in Spotfire. To see that this redundant sentence has been removed, you need to re-publish the Studio project and then go back to Spotfire.

3. Re-publish the Studio project.

Now, in Spotfire, you should see that the redundant sentence does not appear in the narrative when you select the Government segment.

Contextualizing the Narrative

To make the narrative more responsive to selection events within visuals, we’re going to add a prepositional phrase to the first sentence, so we can describe narrowed-down selections in segments. Currently, the first sentence states For the years <years>, the total sales were <sales total> and the total loss was <loss total>. To report on narrowed-down selections, we’re going to add the new phrase after For the years <years>: in the segment <segment name>. An example sentence would look like this (new phrase in bold):

For the years 2018 and 2019, in the segment Enterprise, for the product Montana, the total sales were £2.61MM and the total loss was £31.10K.

1. First, create two new sub-scripts:

segmentPhrase

#define segmentPhrase()[[ segment=getUniqueValues(data,'Segment');
if (len(segment)<=2) {in the [[inflectNoun('segment',len(getUniqueValues(data,'Segment')))]] [[getUniqueValues(data,'Segment')]],}]]

productPhrase

#define productPhrase()[[product=getUniqueValues(data,'Product');
if (len(product)<=2) {for the [[inflectNoun('product',len(getUniqueValues(data,'Product')))]] [[getUniqueValues(data,'Product')]],}]]

 

2. In the Main script, find the year phrase ( [[if (currency!=null){ For the [[inflectNoun('year',len(years))]] [[years]], ...}), and add calls to those two sub-scripts after the comma. The calls to the sub-scripts are shown here in bold:

if (currency!=null){ For the [[inflectNoun('year',len(years))]] [[years]], [[segmentPhrase()]] [[productPhrase()]]

3. Preview the output. You won’t see the new phrase because it only shows up with narrowed-down selections, but as long as you don’t get any errors, the code will work.

4. Re-publish the project, and try it out in Spotfire. With all products selected in the Government segment you should now see:

With only the Paseo product selected in the Government segment you should now see:

 

Going Further

Familiarize yourself with the existing NLG Studio code and try out the Getting Started tutorials. Once you are confident, try to produce the following sentence to add to your starter project narrative. The added sentence describes the sectors having the highest and lowest profitability, for example:

The most profitable segment is Channel Partners (73%) and the lowest is Enterprise (-3%).

If you need help, here is the code for the new sub-script (profitability):

#define profitability() [[sales=groupBy(data,'Segment','Sales','sum');
profit=groupBy(data,'Segment','Profit','sum');
profitabilityBySegments=sort(map(sales,x->(x[0], percentage(filter(profit,y->y[0]==x[0])[0][1],x[1],0,true))), (p,q)->p[1]<=q[1]?1:-1); 
highestProfitable=profitabilityBySegments[0]; lowestProfitable=profitabilityBySegments[len(profitabilityBySegments)-1]; 
if (len(profitabilityBySegments)>1){The most profitable segment is [[highestProfitable[0]]] ([[highestProfitable[1]]]%) and the least is [[lowestProfitable[0]]] ([[lowestProfitable[1]]]%).}]]

 

Here is the call to the new sub-script, which you can place at the end of the paragraph in your Main script:

[[profitability()]]