Skip to main content
waffle.svg
Domo Knowledge Base

ETL 2.0 Tiles: DataSets (Beta)

Version 7

 

Intro

There are three available tiles in the DataSets tab of ETL DataFlows. These include:

  • Input DataSet
  • Output DataSet
  • Writeback tiles

Input and output DataSet tiles are required for all ETL DataFlows. At least one input DataSet is required, though you can include as many as necessary. One output DataSet is required at the end of a DataFlow; this is the DataSet you can then use to power Domo Cards and Apps.  

For information about creating an ETL DataFlow, see Creating an ETL DataFlow.

For information about the Data Center, see Data Center Layout.

Important: Input DataSets in a DataFlow cannot be restricted by PDP policies—all available rows must pass through the DataFlow. Because of this you must apply PDP policies to the output DataSets generated by a DataFlow.
 

When you build a DataFlow using an input DataSet with PDP policies in place, the DataFlow breaks unless at least one of the following criteria applies:

  • You have an "Admin" security profile or a custom role with "Manage DataFlows" enabled.

  • You are the DataSet owner.

  • You are part of the "All Rows" policy. This gives you access to all of the rows in the DataSet.

For more information about using PDP with DataFlows, see PDP and DataFusions/DataFlows

DataSets Tab.png

Getting this feature

If you are interested in joining the Beta for this feature, please contact your Customer Success Manager (CSM).

Input DataSet Tile

Input DataSet Tile.png

You can use the Input DataSet tile to add a DataSet to the DataFlow.

There must be at least one Input DataSet in a DataFlow.

Note: The maximum number of columns allowed in Magic ETL is 1500 columns. 

Configuration

To configure the Input DataSet tile,

  1. Ensure that the data you want to transform already exists in Domo as a DataSet.

  2. Click the Input DataSet tile, then select the DataSet you want to transform.

  3. Set the Data Selection to either be Entire DataSet (default) which will use all of the rows in the DataSet or Only new rows which will only process the newly added rows to the DataSet (this must be done on a DataSet set to Append and not Replace.)

  4. Set the column data types in the Select Data Type dropdown menu next to each column Data Type. This will prevent any schema changes that may happen to the input data from breaking your DataFlow.

  5. Click the ETL Gear Icon.png and set the Special Handling for each column. This will allow you to set the Parsing of the data (only for Number and Date columns), how to handle Bad values, and Null values in your column.

Special Handling Menu Lists.png

Details

Under the Details tab of the input tile, you can view the DataSet's:

  • Name
  • Owner
  • Number of Cards
  • Number of Views
  • Number of Rows
  • Last Updated
  • Next Update
  • Column Names and Types

Data

Select the Data tab to preview a table of the input data.

Output DataSet Tile

Output DataSet Tile.png

You can use the Output DataSet tile to output the transformed data as a DataSet. You can use this new DataSet to power up cards (or other DataFlows).

There must be an Output DataSet in a DataFlow.

Note: Based on scheduled run settings, whenever the specified Input DataSet changes, the ETL Dataflow performs the transform, updating the Output DataSet. For information about scheduling an ETL DataFlow, see Scheduling an ETL DataFlow.

Configuration

To configure the Output DataSet tile,

  1. Ensure that a tile is connected to the Output DataSet tile and that all tiles are connected and configured in the transformation flow.

  2. Click the Output DataSet tile, then specify the name of the DataSet you want to output by clicking  and entering a name and a description.

  3. (Optional) Click the Additional Options dropdown and select the Update Method to either Replace data (default) or Append data.

Tip: You can preview the data in the output DataSet by running a preview, clicking the Output DataSet tile, then clicking the Preview tab.

Details 

If the DataFlow has not run yet, the details available are the owner, column names, and column types. Once run successfully, you are able to view all of the same details listed above for the input tile.

Writeback Tile

Writeback Tile.png

You can use the Writeback tile to output the transformed data as a DataSet and send the data back to another system.

Configuration

To configure the Writeback tile,

  1. Ensure that a tile is connected to the Writeback tile and that all tiles are connected and configured in the transformation flow.

  2. Click the Writeback tile, then specify the name of the DataSet you want to output by clicking  and entering a name and a description.

  3. (Optional) Click the Additional Options dropdown and select the Update Method to either Replace data (default) or Append data.

  4. Click the Configure Writeback button and select the Writeback Connector you will be using.

Details 

If the DataFlow has not run yet, the details available are the owner, column names, and column types. Once run successfully, you are able to view all of the same details listed above for the input tile.