We use cookies. You have options. Cookies help us keep the site running smoothly and inform some of our advertising, but if you’d like to make adjustments, you can visit our Cookie Notice page for more information.
We’d like to use cookies on your device. Cookies help us keep the site running smoothly and inform some of our advertising, but how we use them is entirely up to you. Accept our recommended settings or customise them to your wishes.
×

Frequently Asked Questions about Rapid Audience Layer (RAL)

RAL is available across several cloud platforms including Google Cloud Platform (GCP), Amazon Web Services (AWS) and Microsoft's Azure. The query tools that RAL offers include: Google Big Query, Amazon Redshift, Snowflake and Azure Query Editor.
RAL currently handles flat file data feeds that have fields either in a fixed width format or separated by a delimiter (comma, pipe, tab). In addition, RAL can accept single tab Excel files that have a header row. For RAL clients who select Google Cloud Platform (GCP), there are numerous native Google platforms that can be pushed directly into Big Query using Data Transfer Service (DTS) or Storage Transfer Service (STS).
RAL is able to decompress files that are zipped as well as decrypt files that have been PGP encrypted. We recommend that any feeds that have PII, be transferred as encrypted using Merkle's Public Key (supplied by Merkle RAL team).
Before sending the new feed in the new layout, please email [email protected] and provide details such as the feed name and when the change will be made. The RAL Support team will work with you on making these changes and scheduling when the new feed will be productionalized.
If the feed contains any Pll, clients should not be sending this feed as ad hoc due to recent changes in US and EU laws. For non-Pll feeds, if you have a new set of data you want to push into RAL and it will not be a reoccurring  data source, you can push this feed to RAL by uploading it to the assigned landing location.
RAL's automatic schema detection scans any unknown file it receives to evaluate each column of data and uses AI to assign the most appropriate datatype and stage the file into the database. This is not infallible and bad data in the source feed can throw it off.  For Example: A column of data is called 'product_price' and 99% of the records are sent in a decimal format but 1% of the records are sent with the text "Free". RAL will assign this column a data type of 'string' and not 'float' since there was text detected.
Files should always have a date and time stamp in the filename (YYYYMMDDHHMMSS) and should not have any spaces in the name (Good: Order_Details_20190109110203.txt Bad: Order Details.txt).
Yes. SSH keys are preferred and if a client is sending data via a secure FTP without SSH, there will be a requirement to reset the sFTP password every 60 days.  SSH Keys or approval from client’s Security Officer to remove the password expiration are necessary to relax this requirement.
  • Include a column header
  • Column headers should not have any spaces or special characters. All lowercase characters is highly preferred.
  • Do not use column names that can also be considered SQL commands (Ex: 'date', 'order', 'trim', etc.)
  • When sending a date field, these should be sent as YYYY-MM-DD.
  • If sending date / time stamp values, the following formats are accepted YYYY-MM-DD HH:MM:SS or YYYY-MM-DD HH:MM:SS.MMM
  • Field delimiters can be either pipe, tabs, semi-colons or commas. Be aware of your data values and if some text values have commas in them, choose a different delimiter
  • Record delimiters can be either Carriage Returns and/or Line Feeds
  • It is strongly recommended that you use text qualifiers around your data values, especially if they potentially contain a value that is also serving as the delimiter (Ex: comma, pipe, etc.)
  • NaN and NaT values are not also acceptable as NULL values.
  • A single column file MUST have a header record

What are the methods of sending data to RAL?

Rapid Audience Layer (RAL) provides several ways for flat files to be sent to the platform for ingestion and processing. Some of the different options include:

Using Merkle’s sFTP Server
Using Your Own sFTP Server
Using the Google Cloud SDK (for client's implementing RAL on GCP)
Using the Google Cloud Console (for client's implementing RAL on GCP)

 

Merkle hosts its own sFTP server for file transfers and uses the Globalscape file management platform to poll the sFTP server at set times to look for new data feeds. Globalscape extracts the data, decompresses (.zip, .gz, etc) and decrypts it (if applicable) and pushes the final data into the expected RAL GCP location. Any standard sFTP client can use this technique, either from a command line, shell script, or desktop sFTP client such as FileZilla or CuteFTP.

 

What we need to setup:

  1. Sharing of RAL sFTP credentials and directory location
  2. Expected file name(s) and compression if applicable 
  3. Sharing of Merkle’s RAL Public key for PGP encryption
  4. SSH key (if available)
If you have your own secure FTP (sFTP) server where data feeds can be transferred into, the RAL system can use the Globalscape platform to log into your sFTP server, extract the data, decompress (.zip, .gz, etc.) and decrypt it (if applicable) and push the data feeds into the expected RAL GCP location. Any standard sFTP client can then be used for transferring data into the system, either from a command line command, shell script, or desktop sFTP client such as FileZilla or CuteFTP.

 

What we need to setup:

  1. sFTP credentials and directory location
  2. Expected file name(s) and compression if applicable
  3. Sharing of Merkle’s RAL Public key for PGP encryption
  4. SSH key (if available)
For technical resources who want to move data into RAL, there are tools native to GCP that facilitate automated, scheduled pushes of flat files into the expected GCS bucket.  The GCP Cloud SDK (https://cloud.google.com/sdk/) is the best approach for copying regularly scheduled files to/from GCS via a command on the command line, an application, or a shell script. Some examples of “gsutil” commands can be found here: https://cloud.google.com/storage/docs/renaming-copying-moving-objects
Using the Google Console UI, users 'Upload' files from a user’s computer to be ingested into RAL:  

 

Notes:

  1. The expected directory to push files into is the: /inbound/landing location
  2. This method is good for ad hoc file transfers

Uploading to RAL via the Google Cloud Console (a screenshot)