# Data Integrator

The Data Integrator is the TechWolf feature that eases the process of loading data for both Skills Intelligence and Work Intelligence. It takes the files an organisation delivers, validates and transforms them, and loads the result into the tenant through the SkillEngine API. It is designed for observability and scale: every step produces a visible result, and only what changed is loaded.

## Key attributes

- **Integrations.** Each combination of a source file and an entity type is its own integration, with its own transformation configuration and its own expected delivery schedule. An integration triggers automatically when a new file with its file-name prefix arrives.
- **Transformations.** Source columns are mapped to TechWolf fields, with operations applied where the source format differs: joining files, concatenating fields, parsing dates, mapping values, and generating identifiers.
- **File formats.** Source files are delivered as CSV or Parquet, and can be delivered PGP encrypted for transfer.
- **File validation.** Files are checked before processing: syntactic checks confirm the file is well formed with unique column names, and semantic checks confirm the required columns for the data type are present.

The Data Integrator loads two kinds of data. Entities are the objects TechWolf keeps a record of, and skill events are the evidence employee skill profiles are built from. Custom properties can be loaded on entities alongside either.

| Entity | What the file carries |
| --- | --- |
| Employees | One row per employee, with their assigned position |
| Organisational units | The structure employees are placed in |
| Jobs | The job catalogue, with titles and descriptions |
| Job profile data | Extra job content that does not sit in the source system, in several forms: titles, descriptions, description files, linked vacancies, and skill notes |
| Job families | The grouping above jobs |
| Job family profile data | Extra job family content, as descriptions or description files |
| Job architecture | Jobs and job families delivered in a single file, loaded as separate job and job family integrations rather than each file on its own |
| Vacancies | Open vacancies, which receive their own profile and feed the job's |
| Courses | The learning catalogue, with titles and descriptions |

| Skill event | What the file carries |
| --- | --- |
| Working history | Positions an employee has held, linked to a job in the catalogue or external to it |
| Education | Academic history |
| Certificates | Certifications obtained |
| Resume documents | CVs the organisation holds, as document content in the file or as PDF files referenced from the file |
| Learning | Completed learning, linked to a course in the catalogue or standalone |
| Projects | Projects and completed tasks an employee contributed to |
| Tickets | Work tracked as tickets, with their descriptions |
| Feedback | Performance feedback recorded about an employee |
| Goals | Goals set for or by an employee |
| Skill notes | Skill data recorded outside TechWolf |

The files can come from any system that exports the data. The systems organisations deliver from most often:

| System | What it typically provides |
| --- | --- |
| Learning platforms, such as Cornerstone or SumTotal | The course catalogue and employees' learning history |
| Jira | Tickets: issues, their descriptions and labels |
| ServiceNow | Tickets: incidents, service requests and changes |
| GitHub, GitLab, Bitbucket | Engineering work: pull requests, commits and reviews, from their descriptions and metadata |
| Salesforce | Cases and the day-to-day activity logged against them |
| Confluence | Documentation an employee authored and maintains |
| Project tools, such as Asana or Trello | Projects and completed tasks, with their descriptions |

The list is a starting point rather than a limit: any business application that records attributable work activity can be assessed and added, using a representative sample of the data to confirm the signal it carries first.

## How it works

When a file arrives, the Data Integrator transforms it to TechWolf's data model and updates its record of the desired state: everything that should exist in the tenant according to the source system. It then compares that desired state with what is already loaded, and applies only the differences, creating, updating, and deleting exactly the entities that changed so the tenant mirrors the source system. Failed records are tracked and retried on later runs, so the tenant converges on the source system even when individual records need a dependency that arrives later.

Both full deliveries and incremental deliveries are supported: a full delivery describes the entire dataset each time, while an incremental delivery carries only the changes since the previous file.

## Prerequisites

The Data Integrator is part of TechWolf's input integrations. It requires data files delivered to the tenant's secure file location, on the schedule agreed for each integration.

## Availability

- **Console:** an overview of the integrations in place, their runs and results, and the validation checks and failed records for each file. Integrations and their transformations can be created and edited, runs started or dry-run, and source files delivered, browsed, and managed.