We use cookies to personalize content, to provide social media features and to analyze our traffic. We also share information about your use of our site with our social media, advertising and analytics partners. For information on how to change your cookie settings, please see our Privacy policy. Otherwise, if you agree to our use of cookies, please continue to use our website.

Frequently Asked Questions about Rapid Audience Layer (RAL)

RAL currently handles flat file data feeds that have fields either in a fixed width format or separated by a delimiter (comma, pipe, tab). In addition, RAL can accept single tab Excel files that have a header row. Lastly, because RAL is based on Google Cloud Platform (GCP), there are numerous native Google platforms that can be pushed directly into Big Query using Data Transfer Service (DTS) or Storage Transfer Service (STS).
RAL is able to decompress files that are zipped as well as decrypt files that have been PGP encrypted. We recommend that any feeds that have PII, be transferred as encrypted using Merkle's Public Key (supplied by Merkle RAL team).
RAL can ONLY accept data that uses UTF-8 character encoding.
Before sending the new feed in the new layout, please email [email protected] and provide details such as the feed name and when the change will be made. The RAL Support team will work with you on making these changes and scheduling when the new feed will be productionalized.
If you have a new set of data you want to push into RAL and it will not be a reoccurring data source, you can push this feed to RAL by uploading it in the Google Cloud Storage bucket directory /inbound/landing.  Note: Any data sent to RAL should be directed to the -prod GCP project. If the feed contains any PII, we recommend not sending this feed as ad hoc due to recent changes in US and EU laws.
RAL expands on Big Query's automatic schema detection by scanning the entire file (not just the first 100 rows) to evaluate each column of data and uses AI to assign the most appropriate datatype to assign to get the records to load into Big Query. This is not infallible and bad data in the source feed can throw it off.  For Example: A column of data is called 'product_price' and 99% of the records are sent in a decimal format but 1% of the records are sent with the text "Free". RAL will assign this column a data type of 'string' and not 'float' since there was text detected.
Files should always have a date and time stamp in the filename (YYYYMMDDHHMMSS) and should not have any spaces in the name (Good: Order_Details_20190109110203.txt Bad: Order Details.txt).
Yes. SSH keys are preferred and if a client is sending data via a secure FTP without SSH, there will be a requirement to reset the sFTP password every 60 days.  SSH Keys or approval from client’s Security Officer to remove the password expiration are necessary to relax this requirement.
  • Include a column header
  • Column headers should not have any spaces or special characters. All lowercase characters is highly preferred.
  • Do not use column names that can also be considered SQL commands (Ex: 'date', 'order', 'trim', etc.)
  • When sending a date field, these should be sent as YYYY-MM-DD.
  • If sending date / time stamp values, the following formats are accepted YYYY-MM-DD HH:MM:SS or YYYY-MM-DD HH:MM:SS.MMM
  • Field delimiters can be either pipe, tabs, semi-colons or commas. Be aware of your data values and if some text values have commas in them, choose a different delimiter
  • Record delimiters can be either Carriage Returns and/or Line Feeds
  • It is strongly recommended that you use text qualifiers around your data values, especially if they potentially contain a value that is also serving as the delimiter (Ex: comma, pipe, etc.)
  • NaN and NaT values are not also acceptable as NULL values.
  • A single column file MUST have a header record

What are the methods of sending data to RAL?

Rapid Audience Layer (RAL) is a Google-Cloud based platform and provides several ways for flat files to be sent to the system for ingestion and processing. This document provides details on the following different options for file delivery, including:

  • Using the Google Cloud SDK
  • Using the Google Cloud Console
  • Using Merkle’s sFTP Server
  • Using Your Own sFTP Server
For technical resources who want to move data into RAL, there are tools native to GCP that facilitate automated, scheduled pushes of flat files into the expected GCS bucket.  The GCP Cloud SDK (https://cloud.google.com/sdk/) is the best approach for copying regularly scheduled files to/from GCS via a command on the command line, an application, or a shell script. Some examples of “gsutil” commands can be found here: https://cloud.google.com/storage/docs/renaming-copying-moving-objects
Using the Google Console UI, users 'Upload' files from a user’s computer to be ingested into RAL:  

 

Notes:

  1. The expected directory to push files into is the: /inbound/landing location
  2. This method is good for ad hoc file transfers

Uploading to RAL via the Google Cloud Console (a screenshot)

Merkle hosts its own sFTP server for file transfers and uses the Globalscape file management platform to poll the sFTP server at set times to look for new data feeds. Globalscape extracts the data, decompresses (.zip, .gz, etc) and decrypts it (if applicable) and pushes the final data into the expected RAL GCP location. Any standard sFTP client can use this technique, either from a command line, shell script, or desktop sFTP client such as FileZilla or CuteFTP.

 

What we need to setup:

  1. Sharing of RAL sFTP credentials and directory location
  2. Expected file name(s) and compression if applicable 
  3. Sharing of Merkle’s RAL Public key for PGP encryption
  4. SSH key (if available)
If you have your own secure FTP (sFTP) server where data feeds can be transferred into, the RAL system can use the Globalscape platform to log into your sFTP server, extract the data, decompress (.zip, .gz, etc.) and decrypt it (if applicable) and push the data feeds into the expected RAL GCP location. Any standard sFTP client can then be used for transferring data into the system, either from a command line command, shell script, or desktop sFTP client such as FileZilla or CuteFTP.

 

What we need to setup:

  1. sFTP credentials and directory location
  2. Expected file name(s) and compression if applicable
  3. Sharing of Merkle’s RAL Public key for PGP encryption
  4. SSH key (if available)