The sFTP ICC connector enables you to directly import a dataset from your own server.

Table of contents:

Pre-requisites

Limitations

Creating an sFTP ICC

For some customers having your own sFTP server is a more secure way of enabling the transfer of data without having to rely on third-party storage (eg, AWS s3). sFTP servers have strong cryptographic security (when configured with appropriate key-based authentication) and as a communication standard allow interoperability between various platforms and technology stacks.

Pre-requisites

Cloud Vault

To use your sFTP server for import, you must have a Cloud Vault and add your sFTP details as an ICC (Import Connector Configuration) for that Cloud Vault. You will then be able to reuse the sFTP ICC for any import into your Cloud Vault. 

If you don’t yet have a Cloud Vault, please follow the instructions on this page Importing into your Cloud Vault via the Import Flow – InfoSum

IP Address Authorization

If you are using a VPN or your firewall restricts upload/download of data, you may need to authorize relevant IP addresses.

Limitations

File size limits:

• Single file: 300GB, 18 hours to upload, max single file size
• Multiple files: 1TB, 50gb sharding, 18 hours to upload. Max sharded size.
• Maximum number of files allowed: 5000 (within the above file size limits)

Time limit:

• The maximum duration of an SFTP import is 48 hours. Several variables that affect the speed of a customer’s sFTP server and are outside InfoSum’s control could theoretically push an upload over this limit. The chart below shows details of upload times during testing and is meant to show indicative times. Please speak to your InfoSum representative if you have very large files to understand if another import method might be more suitable.

Creating an sFTP ICC

If you already have created a Cloud Vault, then use the left-hand menu to access the Importing tab.

Then click on the top right button ‘Manage ICC’ and on the next screen click on the ‘Create New ICC’ button. This will take you to the Create New Import Connector Config page.

In this screen, you will add the details for your sFTP server and save them as an ICC.

1. From the Dropdown list of the Connector Type Select the SFTP dropdown

2. Click Next
3. Enter the following information for your sFTP server

• SFTP Hostname or IP Address
• Port 
• Host Public Key (Click on Get Key) and it will find your Host Public Key
• Username
• Password
• Private Key (if applicable)
• Private Key Passphrase (if applicable)

How to find the required keys:

Private Key Pem:

You can ignore this field if you are establishing a connection using a password.

This is a user authorization key (User's SSH private key) replacing the password and will be in the form of a public/private key pair. 

If you are establishing a connection using SSH key/password-less, you will need to add the public ssh key into the authorized keys file on your server and put the private SSH key in the Private Key Pem field in UI.

Host Verification Key:

You will need to enter a host public key in one of the below formats (You need to enter only one)

• Host Public Keys (OpenSSH authorized_keys format) - "Authorized Keys" in Bunker UI
• Host Public Keys (OpenSSH known_hosts format) - "Known Hosts" in Bunker UI
• Host Public Keys (PEM Format)  - "Public Key PEM" in Bunker UI. Currently, we only support PKIX format for public keys. The PEM block with "PUBLIC KEY" will go to this field.

Please note this key is NOT the same as the public part of the user SSH key, this is a public key associated with your server, not with your user.

If clicking on 'Get Key' doesn't work, you can find this key in one of two ways.
1) Your IT team can look it up on the server (probably in the /etc/ssh directory) and there will be a number of files e.g.
     - ssh_host_ecdsa_key.pub
     - ssh_host_ed25519_key.pub
The contents of one of these files can just be put straight into the "Authorized Keys" field on the Bunker UI. An example format for the ecdsa file:

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdH..........<redacted>........LB9u5V+o

2) Alternatively, you can SSH into your server then generate a key pair using the following command "ssh-keygen -F <hostname>" then look up in your local known_hosts file for the public key for that host.

4)  Click Next

5) On the next screen, enter the root directory

The root directory is the location on your sFTP server that is prepended to all file paths defined in your ICC and importer.
Supplying no root directory will cause us to use the default home directory of the user who configured the sFTP server.
A leading "/" denotes a path from the root of the sFTP server, no leading "/" or using "~/" or "./" means from the default home directory.

6) Click Submit.

Once you have entered the sFTP credentials, continue the steps in creating an Import Connector Config.