Hi, How can we help?

GCP Beacons deployment

Beacons is an app for data collaboration that is deployed in your own cloud or warehouse environment, delivering our ‘non-movement of data’ promise even before collaboration has taken place. GCP Beacons are deployed in your Google Cloud BigQuery customer account.

Pre-requisites

BigQuery project setup

Unsupported GCP capabilities

Beacon Deployment

1. Create a GCP Cloud Vault in InfoSum

2. Beacons Marketplace App Deployment in GCP

Next step

Additional information:

Setting up a Beacon for Destination Exports

Deleting your Beacon deployment

Setting up Authorised Views for Cross-Project Access 
 

Pre-requisites

You must have an InfoSum license and account prior to starting this process. If you do not have one, please reach out to your InfoSum contact or support@infosum.com to get started.

You must have your data in a supported format, which is customer-level data. BigQuery can easily transform your data if needed to group the data by customer record. Please review these formatting guidelines and contact our team if you have any questions. 

BigQuery project setup

Each Beacon deployment is one BigQuery dataset and one Cloud Vault.

  • Each Cloud Vault can contain  multiple Beacons/published datasets in InfoSum 
  • Each  BigQuery Dataset  can contain multiple tables and views

This means that you need to ensure that you have a BigQuery Dataset available that contains all your tables and views for collaboration. If your tables are distributed across multiple BigQuery datasets, you have two options:

  1. Deploy two Beacons to two different CV: You will only be able to use the tables on these two Cloud Vaults independently, meaning you cannot publish a joint view of the data to the same Beacon. Please note there is a limit of 10 CVs per account.
  2. Replicate or move the table/view from one dataset to the other: You will then be able to access all tables and views in one Cloud Vault and combine them to create a custom recordset to publish to one single Beacon.
    • If you wish to deploy your Beacon app to a dataset that only contains authorised views please follow these instructions

You must have the following permissions to install an app on your GCP Account. If you don’t have this access please contact your organization administrator.

  • config.deployments.create
  • Config.deployments.update
     

You will also need at least these roles:

  • roles/iam.serviceAccountAdmin
  • roles/iam.serviceAccountUser
  • roles/resourcemanager.projectIamAdmin
     

You must manually enable the "Cloud Resource Manager API" in the GCP project(s) where you plan to install the Beacon app. You can find more information on how to do this in Google’s support documentation for API enablement and the API listing in the marketplace.

(This step is optional and must be completed after app install) If you wish to activate data from your Beacon, you will need to create and manage your own GCP VPC Firewall Rule exception for the network your Beacon has been deployed in to. You do not need to do this if the activation takes place using your partner’s data. Please see this section for more information.

Unsupported GCP capabilities

  • Pure time fields (datetime and date fields are supported)
  • JSON structures 
  • Partitioning strategies and Partition Filters on BigQuery tables
  • We don’t support structured data - (except we use LISTs to hold MULTI-VALUE data. Example below). 

 

Beacon Deployment

There are two distinct tasks to create a GCP Beacon

  1. Create a GCP Cloud Vault
  2. Deploy the Beacon app and link to your Cloud Vault

1. Create a GCP Cloud Vault in InfoSum

The first thing you need to do is create a GCP Cloud Vault in InfoSum that is connected to your BigQuery account. The Cloud Vault won’t host any data; it will simply provide an interface to prepare your data for publishing to your GCP Beacons.

Log in to InfoSum, navigate to the data management section, and click on the Cloud Vault page. 

Then click on the ‘manage cloud vaults’ button on the top right-hand side, and then the ‘create new cloud vault’ button on that page. 

Then you will be asked to select your Cloud Vault provider where your Beacons will be hosted. IN this case please select Google Cloud Vault

 

You will then be asked to give your Cloud Vault a name - this name will identify the data management space if you have multiple. You’ll also need to enter:

  • The region your cloud account is located in. You will need to specify this same zone when deploying the Beacons app to ensure they can connect successfully. The dropdown will show the name of Google regions first and the InfoSum naming convention after (this will be displayed in the Cloud Vault UI).

This will give you a license key that you will need to authenticate in the Beacons app.

Click on finish and connect Cloud Vault to complete this process.

2. Beacons Marketplace App Deployment in GCP

Deploy InfoSum’s Beacons as a marketplace solution deployment into your GCP account. This will link your project with the virtual Cloud Vault on your InfoSum account to manage the collaboration operations.

As part of this process, you will point the InfoSum app to the tables that you wish to use for collaboration. We recommend reviewing our formatting guidelines to ensure that your data can be easily normalised and published to a Beacon.

  1. Find the InfoSum Beacons app in Google’s marketplace
    https://console.cloud.google.com/marketplace/product/infosum-public-460413/infosum.beacons 
  2. Click get started and Agree to Google’s and InfoSum’s terms of service

  3. Click Deploy and configure the app:
    1. Deployment name: Please give a descriptive name that identifies the app in your GCP Project
    2. New vs Existing account: We recommend deploying to a new Service account as this will create an account with the relevant roles that is already set up for you. If you have custom security requirements, you can also deploy using an existing Service Account that has all the appropriate permissions/roles listed on screen.
    3. Service account name: Display name for the service account
    4. Service account ID: It will automatically populate based on the name
    5. Description (optional)
    6. Zone: Please select the same zone that you set up your Cloud Vault account in.
  4. Connect Beacons app to InfoSum
    1. InfoSum Beacons License Key: this is the key collected during the Cloud Vault creation process
    2. Big Query Dataset ID: This is the BigQuery dataset that contains the tables or views that you want to use for collaboration. You can find this inside your BigQuery project, clicking on the three dots next to the dataset.
  1. Click Deploy. The process might take upwards of 15 minutes to complete. You can view more details and information about the deployment in the Details tab (next to Resources)

At the end of the deployment, you will see that there are two InfoSum VM instances running in your compute engine.

You will also see that your GCP cloud vault on the platform has a  "Healthy" status once the connection is established. Please allow for some time for this to complete.

Next step

Once the above steps have been completed successfully, you will find in your Cloud Vault a reference to any tables contained inside the dataset connected to the Beacons app and you will be able to complete the data preparation steps to start collaborating. 

Setting up a Beacon for Destination Exports

This step is optional and only needs to be completed if you wish to export data to a third-party Destination using a Beacon based Dataset. To allow our software to communicate outside of the walls of your GCP account, you must create a Firewall Rule Exception(s) to allow our software to communicate outside of the walls of your GCP account.

You have two options:

  1. Create one exception for all Destinations (recommended): This means that activations are controlled via the InfoSum platform. To complete an export, you will need to set up each Beacon to contain export columns, grant or receive permissions that allow activation, and have an allowed and available Destination to export to.
  2. Create one exception per Destination: If you wish to further restrict the approved Destinations, you can set up your rule exception to point to a specific server or third-party location. You will need to create one rule exception per destination. Activations will still be controlled via the InfoSum platform as described above. 

Please remember you will still need to set up the specific Destination details and access credentials in the InfoSum platform.

Before you get started, check the “Details” tab under your solution deployment. You will need to make a note of two properties: the “Beacon Exports Service Account Email” and the “Network” for your Beacon instance.

Provisioning a Firewall Rule

You may follow one of two methods:

Option A: GCP Web Console

  1. Navigate to “VPC Network” > “Firewall” in your GCP project

  1. Click “Create Firewall Rule” at the top of the page
  2. In the “Create a firewall rule” form that appears:
    1. Enter a name of your choice for the rule
    2. Set the network to the network that you copied from the details tab in the previous section
    3. Enter a priority of 500
    4. Set Direction of traffic to “Egress”
    5. Set Action on match to “Allow”
    6. Set the “Targets” to “Specified service account”
      1. Set the “Target service account” dropdown box that appears to match the “Beacon Exports Service Account Email” output you copied from the details tab in the previous section
    7. Enter an appropriate Destination Filter and IP range. 
  • If you wish to restrict the allowed Destinations to a specific server, for example you may manage your own SFTP server with a specific public IP address, in which case you should enter only the IP of the server you wish the Beacon to export to. 
    • If unsure about which network range to allow here, please consult a relevant Google Cloud or network administrator within your Organization
  • If you wish to create one rule for all Destinations, please set a range of 0.0.0.0/0 

 

4. Once this is done you should create the rule, then confirm that the rule is present in the Firewall Rules list, and the details (network etc) match the ones for your Beacon

Example of network configuration

Option B: Google Cloud CLI

You can use the official Google Cloud CLI tool to manage your Firewall Rule instead. Please run the following command in your terminal or via cloud shell in the GCP console, replacing the relevant properties with the ones you copied from the details tab in the previous section:

  • Replace <PROJECT> with your GCP Project ID. You can find this in the homepage of your GCP project.
  • Replace <NETWORK> with the Beacon’s network
  • Replace <SERVICE_ACCOUNT> with the Beacon’s service account email you copied
  • Replace <RANGES> with a comma separated list of network ranges you wish to export to
    • If you wish to restrict the allowed Destinations to a specific server, for example you may manage your own SFTP server with a specific public IP address, in which case you should enter only the IP of the server you wish the Beacon to export to. 
      If unsure about which network range to allow here, please consult a relevant Google Cloud or network administrator within your Organization
    • If you wish to create one rule for all Destinations, please set a range of 0.0.0.0/0 

gcloud config set project <PROJECT>

gcloud compute firewall-rules create allow-push-egress-to-all \

  --network=<NETWORK> \

  --direction=EGRESS \

  --priority=500 \

  --action=ALLOW \

  --rules=all \

  --destination-ranges=<RANGES> \

  --target-service-accounts=<SERVICE_ACCOUNT> \

  --description="Beacon egress exception rule for destination exports”

Deleting your Beacon deployment

If you have deployed a Beacon to the wrong BQ Dataset, some of the settings are incorrect or you no longer need the Beacon, you will need to uninstall it. There are two steps to do so: remove the compute engine VM instances and then delete the deployment. You can do this on your own or reach out to your InfoSum representative who will be happy to help guide you through the steps

  1. Delete all Compute Engine VM instances in your Beacon’s network.
    This includes any additional VMs that may have been launched since you initially deployed your Beacon instance
    1. These instances may be identified by filtering on the Beacon network in the Compute Engine “VM Instances” page, using the table column filter
    2. You may find several VMs within the Beacon’s network, with names that may or may not include “bootstrap”, “control-container”, “psh” or “rbi”. All of these must be deleted via the console or GCP API

      Important: Wait until the VMs have been fully deleted and are no longer present in the VM Instances list in your project.
  2. Delete all custom Firewall Rules in your Beacon’s network
    1. This includes any rules you may have created to allow egress to external networks to allow exports to Destinations
  3. Delete the Beacon Deployment: Once this is complete, navigate to the “Solution Deployment” page under your GCP Console, and identify the instance of the beacon you wish to uninstall using the deployment name.
    1. Click the three dots next to the deployment, and click “Delete”. Confirm the option, then wait for the resources to be destroyed. This may take some time.

Setting up Authorised Views for Cross-Project Access 

Authorised Views in BigQuery let us securely share specific query results across projects without exposing the underlying tables. They act as a controlled window into the data, allowing access only to the fields and rows defined in the view. In this setup, we use authorised views to provide the InfoSum Beacon with limited, read-only access to approved datasets, ensuring least-privilege data sharing and compliance with Google’s best practices.

Pre-requisites

  • You have an existing dataset in BigQuery that contains at least one table with the data you want to share.
  • You have access to two GCP projects:
    • Source Project - holds the original dataset/table.
    • Destination Project - where the InfoSum Beacon will be deployed
  • You have appropriate IAM roles (e.g. roles/bigquery.dataOwner or roles/bigquery.admin) in both projects.
     
  1. Create the query that defines the data to share

In the source project, open BigQuery and write the query that selects the data you want to expose to the InfoSum Beacon App.

Examples:

-- Share all data
SELECT * FROM `source_project.source_dataset.source_table`
-- Share filtered data
SELECT name, region, total_sales
FROM `source_project.source_dataset.source_table`

2. Save the query as a View in a new dataset

Create a separate dataset (in your source project) to hold the authorised views.

  • Select your current project
  • Create a new dataset to hold your authorised view (referenced on this guide as shared dataset)
  • The table field will be the name for your new view (referenced on this guide as shared view)

Your view reference is: shared_dataset.shared_view

This follows Google’s best-practice guidance: separate authorised views from the source data for clarity and security 

3. Authorise the new view to access the source dataset

In the console:

  1. Open source dataset → Share → Authorize Views.
  2. Click Add Authorisation → Select your new view (shared_dataset.shared_view).

4. Grant access to the principal (service account / user)

Give the principal that will query the view permission to read from the shared dataset (not the source one). In the source project:

  • Open the shared_view view (the one containing your authorised view).
  • Add the principal (What user/SA will be used for querying the authorised view) with:
    roles/bigquery.dataViewer

This gives them access to query the view without direct access to the raw data.

5. Create new View in Destination project

Now from the destination project, using the authorised principal, you can query:

SELECT * FROM `source_project.dataset_shared.sales_view`

If access is correctly configured, this will return the filtered dataset defined by your view - even though the raw source table is private.

This view will query and return all results from the authorised view in the source project. You then use the dataset ID that contains this view for the beacon app installation. 

You can now continue with the Beacon app installation

Was this article helpful?

Have more questions? Submit a request

-