Accelerating innovation in education: Implementing the Prometheus-X Data Space Connector on AWS

The education sector is experiencing unprecedented digital transformation, driving the need for secure, efficient data exchange between organizations. Amazon Web Services (AWS) offers a comprehensive suite of services that help educational institutions, EdTech companies, and learning providers navigate this transformation while maintaining the highest standards of data security and compliance. In this post, we’ll explore an architecture to facilitate data transfer, and we will demonstrate how to implement the Prometheus-X Data Space Connector (PDC) on AWS. This solution enables organizations to participate in the education and skills data ecosystem securely and efficiently.

Prometheus-X (PTX) is a pioneering initiative that establishes a data space dedicated to the education and skills domain, enabling organizations to share and access data in a controlled and standardized manner. In a world where data is the new currency, it’s essential to have data spaces that guarantee trust, interoperability, and governance. For companies, it’s an opportunity to collaborate with others without compromising data integrity. For institutions, it’s a way to provide ethical and fair access to information. For citizens, it’s a guarantee that their data won’t be used without their permission and consent.

The skills and education data space creates opportunities across the educational technology sector. EdTech companies can participate either as data providers, monetizing and sharing their collected data, or as service providers who enhance their offerings by accessing high-quality educational data through the platform.

Edtake, a French EdTech company, leveraged the data space to create an AI-powered learning design system. By integrating data from Inokufu (another EdTech), including course syllabi, skills frameworks, and existing learning materials, Edtake streamlined their content creation process. This collaboration enables more efficient learning object development, skills mapping, and content reuse, demonstrating the practical benefits of data sharing in educational technology. Data spaces provide a secure and standardized environment that makes such seamless data sharing and collaboration possible between EdTech companies.

EdTech companies can join the skills and education data space to unlock multiple opportunities. Organizations gain access to innovative services already integrated within the ecosystem and enhance their product offerings with cutting-edge technologies for their clients. The data space enables EdTechs to monetize their data by selling it to innovative service providers and industry leaders. Companies can access high-quality data in a secure, compliant environment, leverage existing market services to enhance their products, and reduce technical integration costs.

To facilitate seamless integration into this data space, the Prometheus-X Data Space Connector (PDC) serves as a bridge between an organization’s internal systems and the PTX ecosystem. This post provides a comprehensive walkthrough on setting up the PDC on AWS, empowering organizations to participate in the skills and education data space effectively.

Understanding data spaces and the role of the connector

Data spaces are collaborative frameworks that allow multiple organizations to share data securely and efficiently while maintaining control over their own data assets. They’re designed to foster interoperability, data sovereignty, and trust among participants.

In a data space, connectors are essential components that enable organizations to interface with the shared ecosystem. They handle the complexities of data exchange protocols, authorization, and compliance, allowing participants to focus on their core operations without the need to develop custom integration solutions.

The education and skills sectors involve a diverse range of stakeholders, including educational institutions, training providers, employers, and learners. A standardized connector like the PDC ensures that these actors can exchange data seamlessly, supporting use cases such as credential verification, course enrollment, and skills assessment.

Prometheus-X is an initiative aimed at creating a human-centric and sustainable data space for education and skills. It provides the necessary infrastructure and protocols to enable secure data sharing among various stakeholders, fostering innovation and collaboration in the sector.

Solution overview

This section outlines the dataspace architecture where the PDC runs on Amazon Elastic Container Service (ECS) by using AWS Fargate. The architecture demonstrates how data assets are shared and processed through the system.

The PDC connector enables secure sharing of data assets, which are initially stored in a raw Amazon Simple Storage Service (Amazon S3) bucket. When shared, these assets trigger an AWS Glue job that cleans the data and standardizes its format, storing the results in a destination S3 bucket. Finally, an AWS Lambda function registers these data assets in the catalog, after processing them with AWS Glue. Amazon Route 53 manages the domain name required by VisionsTrust and sends the requests to an Application Load Balancer, with a certificate managed by AWS Certificate Manager to enable secure SSL/TLS connection. This domain managed by Route 53 is where the information from your connector resides.

The processed shared assets can then be utilized by various AWS analytics and AI services. Through Amazon Athena and Amazon QuickSight, users can create interactive business intelligence dashboards. Finally, Amazon SageMaker AI enables machine learning workloads, with the data assets that have been securely shared and facilitated by the PDC connector.

In conclusion, once the shared assets reside in AWS, they can be transformed through an ETL pipeline and either visualized using Amazon QuickSight or leveraged for machine learning applications with SageMaker AI, providing organizations with comprehensive data processing and analysis capabilities.

Figure 1. AWS architecture

PDC is an open-source project designed to simplify the onboarding and participation of organizations in the data space. It offers the necessary functionalities for organizations to communicate with core components of the data space, such as the catalogue for resource management, contract for negotiation and verification, and consent for managing consent-driven data exchanges.

Data exchange within a data space involves the transfer of data between participants under agreed-upon terms and conditions. The PDC facilitates this process by:

• Interacting with the catalogue, contract, and consent services – The connector manages interactions with these core services, ensuring that data exchanges are conducted according to established agreements and policies.

• Managing authorizations and control plane verifications – It handles authorization processes and verifies control plane operations related to data exchanges, enabling compliance with the ecosystem’s standards.

• Conforming to data space protocols – The PDC adheres to data space protocols set by PTX, eliminating the need for organizations to reimplement each API in the data space core services.

The following diagram illustrates the interaction between an organization’s IT services and the Prometheus-X services through the PDC. In this setup, the organization’s IT services communicate with the PDC, which in turn interfaces with the Prometheus-X core services—facilitating secure and compliant data exchanges.

When implementing the PDC on AWS for education data, ensure you follow these security best practices.

Figure 2: Interaction between dataspace participants and intermediaries

Setting up the PDC on AWS

These steps demonstrate how to set up the PDC on an EC2 instance with Docker (installed via a user script) on AWS. For production-grade deployments, we recommend using an orchestration service, such as ECS or EKS (where multiple containers can be deployed), as discussed in the previous section.

Prerequisites

To set up the PDC on AWS, you need to have the following prerequisites in place:

1. An active AWS account.
2. AWS Identity and Access Management (IAM) with appropriate permissions to access the AWS services listed in the architecture.

VisionsTrust is an implementation of Prometheus-X services that provides the necessary infrastructure for participants in the data space.

1. Visit VisionsTrust to create your account by providing your organization’s information.
2. Complete the registration by submitting details about your organization, such as name, address, and contact information.
3. API keys for your participant are automatically generated. Retrieve them by visiting the profile section of your account and choose the API tab.

Set up an Amazon Elastic Compute Cloud (Amazon EC2) instance on AWS:

1. To launch an EC2 instance, on the Amazon EC2 console, select an appropriate AMI such as Amazon Linux 2, which is compatible with Docker.
2. To configure instance details, specify the instance type, network settings, and storage options according to your requirements.
3. To add a user data script, in the Advanced details section, add the following user data script to automate the installation of Docker, the PDC, and Nginx, replacing your-domain.com with your chosen domain and the credentials “serviceKey” and “secretKey” in the configuration file with the API keys obtained from VisionsTrust. Finally, your-endpoint.com is the endpoint available in the VisionsTrust dashboard. The user script below installs the PDC on your EC2.

   #!/bin/bash# Update the system
   
   yum update -y
   
   # Install necessary packages
   
   sudo yum install docker -y
   
   sudo systemctl start docker
   
   sudo usermod -a -G docker ec2-user
   
   # Install Docker Compose
   
   sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
   
   chmod +x /usr/local/bin/docker-compose
   
   # Install Git
   
   yum install git -y
   
   # Clone the PDC repository
   
   cd /home/ec2-user
   
   git clone https://github.com/Prometheus-X-association/dataspace-connector.git
   
   cd dataspace-connector
   
   # Set up configuration file
   
   cat > src/config.production.json <<EOL
   
   {
   
                   "endpoint": "your-endpoint.com",
   
                   "catalogUri": "https://api.visionstrust.com/v1/",
   
                   "contractUri": "https://contract.visionstrust.com/",
   
                   "consentUri": "https://consent.visionstrust.com/v1",
   
                   "serviceKey": "your service key found at https://visionstrust.com/dashboard/profile/settings",
   
                   "secretKey": "your secret key found at https://visionstrust.com/dashboard/profile/settings"
   
   }
   
   EOL
   
   # Update the src/config.production.json file with your specific configurations as needed
   
   # Setup env variables
   
   sudo cp .env.sample .env
   
   # Adjust the env variables to your configuration
   
   # Build and run the Docker containers
   
   sudo docker-compose build
   
   sudo docker-compose up -d
   
   # Install Nginx
   
   sudo yum install nginx -y
   
   sudo systemctl start nginx
   
   # Configure Nginx as a reverse proxy
   
   sudo tee /etc/nginx/conf.d/pdc.conf > /dev/null <<EOL
   
   server {
   
       listen 80;
   
       server_name your_domain.com;
   
       location / {
   
           proxy_pass http://localhost:3000;
   
           proxy_set_header Host \$host;
   
           proxy_set_header X-Real-IP \$remote_addr;
   
           proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
   
       }
   
   }
   
   EOL
   
   # Restart Nginx to apply the configuration
   
   sudo systemctl restart nginx

4. In the security group configuration, for the inbound rules, confirm that the security group associated with the EC2 instance allows inbound traffic on ports 80 (HTTP) and 443 (HTTPS) to enable web access.
5. Assign a domain name. If you have a domain name, configure it to point to the public IP address of your EC2 instance. You can use Amazon Route 53 for this. You can also use an Elastic IP address; assign it to the EC2 instance you launched and replace your_domain.com by its IPv4 address.
6. Make sure there is a config.json file in your project. If you cannot find it, create one and replace it with your own secret key, domain and service key.
7. Finally, you can add secure communication, by setting up SSL or TLS certificates by using AWS Certificate Manager or other free alternatives (for example Let’s Encrypt). We recommend you set up SSL/TLS.

To verify your setup:

1. Go to https://your-domain.com. This should display JSON information about your connector, indicating that the setup was successful. Another alternative is to use a native AWS service such as AWS Certificate Manager (ACM) to manage the certificate and Amazon Route 53 as the DNS service. At a high level, you need to create a public hosted zone in Route 53 and then you can update the NS/A record to direct the traffic to the EC2 instance. You can check if the record was mapped correctly by pinging your domain.
2. You need to request an SSL certificate in ACM by using the console or the AWS Command Line Interface (AWS CLI). For more information, refer to request-certificate.
3. To verify domain ownership in Route 53, check the status. After the domain has been validated, the status will change from Pending validation to Available.

These steps can also be carried out using the AWS CLI directly.

Conclusion

In this post, we presented an architecture to run the PDC on AWS and an event-driven architecture to clean the data and store it in a destination S3 bucket, once new data has been detected. Furthermore, by following this guide, you can successfully set up the Prometheus-X Data Space Connector on an EC2 instance, enabling your organization to participate in the skills and education data space. This integration opens possibilities to connect with other actors in the ecosystem through a shared set of protocols and standards covered by the connector.

For more detailed information and advanced configurations, refer to the official Prometheus-X Data Space Connector documentation. Get started today by visiting the AWS Console!