Deploying WDS API Server in Docker Compose

Recommended for evaluation and development: run the API Server plus dependencies locally via Compose. The API binds to port 2807 at http://localhost:2807.

To make use of this deployment type, docker-compose needs to be installed locally. Use the following command to check if it’s already installed:

docker compose version

the result should be like this:

Docker Compose version <version>

If the result shows that docker-compose is not installed, please review the installation documentation on its official website

Deployment Options

Taking into account that the API Server requires third-party components, there are several deployment options:

MINI (Free) - runs solidstack, and auxiliary services services in docker-compose
BOX (Free) - runs solidstack, auxiliary services services, and third-party components in docker-compose
OEM (Enterprise) - runs core services stack, and auxiliary services services in docker-compose
BOX (Enterprise) - runs core services stack, auxiliary services services, and third-party components in docker-compose

This deployment option contains all third-party services for system evaluation, even in an air-gapped environment.

IMPORTANT! MongoDB connection string must contain a database name. It might be Solidstack as in the example connection string or anything else.


services:
  mongodb-primary:
    image: docker.io/zcube/bitnami-compat-mongodb:6.0
    restart: always
    hostname: mongodb-primary
    environment:
      - MONGODB_ADVERTISED_HOSTNAME=mongodb-primary
      - MONGODB_ROOT_USER=root
      - MONGODB_ROOT_PASSWORD=TestPassword
      - MONGODB_REPLICA_SET_MODE=primary
      - MONGODB_REPLICA_SET_KEY=TestReplicasetKey
  
  mongodb-secondary:
    image: docker.io/zcube/bitnami-compat-mongodb:6.0
    restart: always
    hostname: mongodb-secondary
    depends_on:
      - mongodb-primary
    environment:
      - MONGODB_ADVERTISED_HOSTNAME=mongodb-secondary
      - MONGODB_REPLICA_SET_MODE=secondary
      - MONGODB_INITIAL_PRIMARY_HOST=mongodb-primary
      - MONGODB_REPLICA_SET_KEY=TestReplicasetKey
      - MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD=TestPassword

  mongodb-arbiter:
    image: docker.io/zcube/bitnami-compat-mongodb:6.0
    restart: always
    hostname: mongodb-arbiter
    depends_on:
      - mongodb-primary
    environment:
      - MONGODB_ADVERTISED_HOSTNAME=mongodb-arbiter
      - MONGODB_REPLICA_SET_MODE=arbiter
      - MONGODB_INITIAL_PRIMARY_HOST=mongodb-primary
      - MONGODB_REPLICA_SET_KEY=TestReplicasetKey
      - MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD=TestPassword
 
  wds.solidstack:
    image: docker.io/webdatasource/solidstack:v2.0.3
    restart: always
    hostname: solidstack
    environment:
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - JOB_TYPES=intranet
      - DB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb-primary,mongodb-secondary/Solidstack?authSource=admin&replicaSet=replicaset&readPreference=secondary
    ports:
    - 2807:8080
  
  wds.playground:
    image: docker.io/webdatasource/playground:v2.0.3
    restart: always
    hostname: playground
    ports:
    - 2808:80
  
  wds.docs:
    image: docker.io/webdatasource/docs:v2.0.5
    restart: always
    ports:
    - 2809:80

This deployment option implies connecting to an external MongoDB. For evaluation purposes, a Free MongoDB Atlas cluster can be used.
In the wds.solidstack service replace the value of the DB_CONNECTION_STRING environment variable with a real connection string to a MongoDB database.

IMPORTANT! MongoDB connection string must contain a database name. It might be Solidstack as in the example connection string or anything else.


services:  
  wds.solidstack:
    image: docker.io/webdatasource/solidstack:v2.0.3
    restart: always
    hostname: solidstack
    environment:
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - JOB_TYPES=intranet
      - DB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Solidstack?appName=<cluster>&readPreference=secondary
    ports:
      - 2807:8080
  
  wds.playground:
    image: docker.io/webdatasource/playground:v2.0.3
    restart: always
    hostname: playground
    ports:
    - 2808:80
  
  wds.docs:
    image: docker.io/webdatasource/docs:v2.0.5
    restart: always
    ports:
    - 2809:80

This deployment option implies connecting to an external MongoDB. For evaluation purposes, a Free MongoDB Atlas cluster can be used.
In the wds.dapi, wds.datakeeper, and wds.idealer services replace values of the DB_CONNECTION_STRING environment variables with real connection strings to a MongoDB database.

By default, system DB (MongoDB) is used to cache web pages. This behavior can be changed by providing the wds.datakeeper service with a CACHE_CONNECTION_STRING.

IMPORTANT! MongoDB connection strings must contain a database name. The names might be the same as values in the example connection strings or others. IMPORTANT! This feature is available starting from the Business Plan


services: 
  wds.crawler:
    image: docker.io/webdatasource/crawler:v2.0.3
    restart: always
    hostname: crawler
    environment:
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SERVICE_HOST=crawler
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - LICENSE_KEY=[LICENSE]
  
  wds.datakeeper:
    image: docker.io/webdatasource/datakeeper:v2.0.3
    restart: always
    hostname: datakeeper
    environment:
      - DB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Datakeeper?appName=<cluster>&readPreference=secondary
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]

  wds.idealer:
    image: docker.io/webdatasource/idealer:v2.0.3
    restart: always
    hostname: idealer
    environment:
      - DB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Idealer?appName=<cluster>&readPreference=secondary
      - LICENSE_KEY=[LICENSE]
  
  wds.scraper:
    image: docker.io/webdatasource/scraper:v2.0.3
    restart: always
    hostname: scraper
    environment:
      - LICENSE_KEY=[LICENSE]
  
  wds.dapi:
    image: docker.io/webdatasource/dapi:v2.0.3
    restart: always
    hostname: dapi
    environment:
      - DB_CONNECTION_STRING=mongodb+srv://<user>:<password>@<host>/Dapi?appName=<cluster>&readPreference=secondary
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - IDEALER_ORIGIN=http://idealer
      - JOB_TYPES=intranet
      - LICENSE_KEY=[LICENSE]
    ports:
      - 2807:8080
  
  wds.playground:
    image: docker.io/webdatasource/playground:v2.0.3
    restart: always
    hostname: playground
    ports:
    - 2808:80
  
  wds.docs:
    image: docker.io/webdatasource/docs:v2.0.5
    restart: always
    ports:
    - 2809:80

This deployment option contains all third-party services for system evaluation, even in an air-gapped environment.
There is only one instance per service, and credentials are not protected since this is an evaluation environment.

IMPORTANT! This feature is available starting from the Business Plan


services:
  mongodb-primary:
    image: docker.io/zcube/bitnami-compat-mongodb:6.0
    restart: always
    hostname: mongodb-primary
    environment:
      - MONGODB_ADVERTISED_HOSTNAME=mongodb-primary
      - MONGODB_ROOT_USER=root
      - MONGODB_ROOT_PASSWORD=TestPassword
      - MONGODB_REPLICA_SET_MODE=primary
      - MONGODB_REPLICA_SET_KEY=TestReplicasetKey
  
  mongodb-secondary:
    image: docker.io/zcube/bitnami-compat-mongodb:6.0
    restart: always
    hostname: mongodb-secondary
    depends_on:
      - mongodb-primary
    environment:
      - MONGODB_ADVERTISED_HOSTNAME=mongodb-secondary
      - MONGODB_REPLICA_SET_MODE=secondary
      - MONGODB_INITIAL_PRIMARY_HOST=mongodb-primary
      - MONGODB_REPLICA_SET_KEY=TestReplicasetKey
      - MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD=TestPassword

  mongodb-arbiter:
    image: docker.io/zcube/bitnami-compat-mongodb:6.0
    restart: always
    hostname: mongodb-arbiter
    depends_on:
      - mongodb-primary
    environment:
      - MONGODB_ADVERTISED_HOSTNAME=mongodb-arbiter
      - MONGODB_REPLICA_SET_MODE=arbiter
      - MONGODB_INITIAL_PRIMARY_HOST=mongodb-primary
      - MONGODB_REPLICA_SET_KEY=TestReplicasetKey
      - MONGODB_INITIAL_PRIMARY_ROOT_PASSWORD=TestPassword
  
  minio:
    image: docker.io/bitnami/minio:2024
    restart: always
    hostname: minio
    environment:
      - MINIO_DEFAULT_BUCKETS=pages-cache
      - MINIO_ROOT_USER=TestAccessKey
      - MINIO_ROOT_PASSWORD=TestSecretKey
      - MINIO_FORCE_NEW_KEYS=no
 
  wds.crawler:
    image: docker.io/webdatasource/crawler:v2.0.3
    restart: always
    hostname: crawler
    environment:
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SERVICE_HOST=crawler
      - EXTERNAL_IP_ADDRESS_CONFIGS=intranet
      - LICENSE_KEY=[LICENSE]
  
  wds.datakeeper:
    image: docker.io/webdatasource/datakeeper:v2.0.3
    restart: always
    hostname: datakeeper
    environment:
      - DB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb-primary,mongodb-secondary/Datakeeper?authSource=admin&replicaSet=replicaset&readPreference=secondary
      - CACHE_CONNECTION_STRING=s3://TestAccessKey:TestSecretKey@minio:9000/pages-cache?ssl=false
      - IDEALER_ORIGIN=http://idealer
      - LICENSE_KEY=[LICENSE]

  wds.idealer:
    image: docker.io/webdatasource/idealer:v2.0.3
    restart: always
    hostname: idealer
    environment:
      - DB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb-primary,mongodb-secondary/Idealer?authSource=admin&replicaSet=replicaset&readPreference=secondary
      - LICENSE_KEY=[LICENSE]
  
  wds.scraper:
    image: docker.io/webdatasource/scraper:v2.0.3
    restart: always
    hostname: scraper
    environment:
      - LICENSE_KEY=[LICENSE]
  
  wds.dapi:
    image: docker.io/webdatasource/dapi:v2.0.3
    restart: always
    hostname: dapi
    environment:
      - DB_CONNECTION_STRING=mongodb://root:TestPassword@mongodb-primary,mongodb-secondary/Dapi?authSource=admin&replicaSet=replicaset&readPreference=secondary
      - DATAKEEPER_ORIGIN=http://datakeeper
      - SCRAPER_ORIGIN=http://scraper
      - IDEALER_ORIGIN=http://idealer
      - JOB_TYPES=intranet
      - LICENSE_KEY=[LICENSE]
    ports:
      - 2807:8080
  
  wds.playground:
    image: docker.io/webdatasource/playground:v2.0.3
    restart: always
    hostname: playground
    ports:
    - 2808:80
  
  wds.docs:
    image: docker.io/webdatasource/docs:v2.0.5
    restart: always
    ports:
    - 2809:80ways
    ports:
    - 2809:80

Running Docker Compose

After an appropriate deployment option is selected and a docker-compose configuration has been copied to a file (e.g., wds-docker-compose.yml), there are two types of scripts to run docker-compose:

Windows - for Windows OS
Linux - for Linux-based OS including MacOS

NOTE_ If another version is currently running, just replace the old docker-compose configuration with the new one (of the same option) and execute the selected script. The only services with changed versions will be recreated. If the new version is of a different option, execute the docker compose down command beforehand.

set COMPOSE_FILE=wds-docker-compose.yml && docker compose up -d

export COMPOSE_FILE=wds-docker-compose.yml && docker compose up -d

Deploying WDS API Server in Docker Compose

Deployment Options

Running Docker Compose

Please rotate your device to landscape mode