Reproducing Gitlab CI failures in Docker Compose locally

OK, this is a fun situation.

A piece of code works and passes tests locally (local dev environment is a Vagrant) but fails the Drupal install phase in Gitlab CI runner.

The failure is that in the Gitlab CI job, drush site-install task bails out with:

drush site:install -y --db-url=pgsql://example:example@db/example --existing-config

 You are about to:
 * DROP all tables in your 'example' database.

 // Do you want to continue?: yes.                                                                                 

 [notice] Starting Drupal installation. This takes a while.
 [notice] Performed install task: install_select_language
 [notice] Performed install task: install_select_profile
 [notice] Performed install task: install_load_profile
 [notice] Performed install task: install_verify_requirements
 [notice] Performed install task: install_settings_form
 [notice] Performed install task: install_write_profile
 [notice] Performed install task: install_verify_database_ready
 [notice] Performed install task: install_base_system
 [notice] Performed install task: install_bootstrap_full
 [warning] The "block_content:b42edb07-dcb6-4be4-aa3b-480fc6e06ccf" was not found
 [notice] Options titles do not exist.
 [notice] Options state_province_names do not exist.

In FieldConfig.php line 312:
                                                                                                
  Attempt to create a field field_not_quite_logged_in that does not exist on entity type user.  
                                                                                                

site:install [--db-url DB-URL] [--db-prefix DB-PREFIX] [--db-su DB-SU] [--db-su-pw DB-SU-PW] [--account-name [ACCOUNT-NAME]] [--account-mail [ACCOUNT-MAIL]] [--site-mail [SITE-MAIL]] [--account-pass ACCOUNT-PASS] [--locale [LOCALE]] [--site-name [SITE-NAME]] [--site-pass SITE-PASS] [--sites-subdir SITES-SUBDIR] [--config-dir CONFIG-DIR] [--existing-config] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-USER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--notify] [--druplicon] [--xh-link XH-LINK] [--] <command> [<profile>]...

I tried to reproduce this locally in the existing Vagrant environment, and it did not appear there; drush site-install completed as expected.

I emulate the Gitlab CI environment (which uses a Docker CI runner environment) using Docker Compose to reproduce the failure. Here's the Gitlab CI yaml snippet:

# Reduced version of https://gitlab.example.org/example/example/-/blob/<hash-here>/.gitlab-ci.yml
image: gitlab.example.org:4567/drupal/docker/php:7.2-latest

test behat:
  stage: test
  needs:
    - build theme
    - build drupal
  tags:
    - docker
  services:
    - name: postgres:10.12
      alias: db
  variables:
    HOST_PORT: "0.0.0.0:8888"
    DRUPAL_ENVIRONMENT: ci
    POSTGRES_DB: example
    POSTGRES_USER: example
    POSTGRES_PASSWORD: example
  before_script:
    - echo '<?php $environment = getenv("DRUPAL_ENVIRONMENT");' > web/sites/default/settings.local.php
    - drush site:install -y --db-url=pgsql://example:example@db/example --existing-config
    - php -S $HOST_PORT -t ./web &> tmp/php.log.txt &
    - until curl -s http://$HOST_PORT; do true; done > /dev/null
    - curl -o tmp/index.html -s http://$HOST_PORT
    - cd tests/behat
  script:
    - behat --profile=gitlab_ci --tags='~@broken&&~@drush' -f junit -o $CI_PROJECT_DIR/behat-junit -f pretty -o std --colors

Here's the docker-compose.yml I use to reproduce what the above will do:

version: "3.1"
services:
  db:
    image: postgres:10.12
    environment:
        POSTGRES_DB: example
        POSTGRES_USER: example
        POSTGRES_PASSWORD: example
  app:
    image: gitlab.example.org:4567/drupal/docker/php:7.2-latest
    environment:
        HOST_PORT: "0.0.0.0:8888"
        DRUPAL_ENVIRONMENT: ci
    working_dir: /app
    command: ['php', '-S', '0.0.0.0:8888', '/app/web']
    volumes:
        - ./:/app

I then need to reproduce the behaviour of the Gitlab CI before_script and script steps.

I set the environment in settings.local.php:

/app # echo '<?php $environment = getenv("DRUPAL_ENVIRONMENT");' > web/sites/default/settings.local.php

Then I execute the step which fails:

/app # drush site:install -y --db-url=pgsql://example:example@db/example --existing-config

 You are about to:
 * DROP all tables in your 'example' database.

 // Do you want to continue?: yes.                                                                                 

 [notice] Starting Drupal installation. This takes a while.
 [notice] Performed install task: install_select_language
 [notice] Performed install task: install_select_profile
 [notice] Performed install task: install_load_profile
 [notice] Performed install task: install_verify_requirements
 [notice] Performed install task: install_settings_form
 [notice] Performed install task: install_write_profile
 [notice] Performed install task: install_verify_database_ready
 [notice] Performed install task: install_base_system
 [notice] Performed install task: install_bootstrap_full
 [warning] The "block_content:b42edb07-dcb6-4be4-aa3b-480fc6e06ccf" was not found
 [notice] Options titles do not exist.
 [notice] Options state_province_names do not exist.

In FieldConfig.php line 312:
                                                                                                
  Attempt to create a field field_not_quite_logged_in that does not exist on entity type user.  
                                                                                                

site:install [--db-url DB-URL] [--db-prefix DB-PREFIX] [--db-su DB-SU] [--db-su-pw DB-SU-PW] [--account-name [ACCOUNT-NAME]] [--account-mail [ACCOUNT-MAIL]] [--site-mail [SITE-MAIL]] [--account-pass ACCOUNT-PASS] [--locale [LOCALE]] [--site-name [SITE-NAME]] [--site-pass SITE-PASS] [--sites-subdir SITES-SUBDIR] [--config-dir CONFIG-DIR] [--existing-config] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-d|--debug] [-y|--yes] [--no] [--remote-host REMOTE-HOST] [--remote-user REMOTE-USER] [-r|--root ROOT] [-l|--uri URI] [--simulate] [--pipe] [-D|--define DEFINE] [--notify] [--druplicon] [--xh-link XH-LINK] [--] <command> [<profile>]...

This fails, which is a great success! From here, I debugged a little further to establish it was something like a timing issue on first run. Those details ended up in the related client-specific issue, and I kept these notes for sharing here.

The real takeaway for me was that I could readily use Docker Compose to replicate issues I saw in Gitlab CI, and knowing that a significant amount of variation between environments could be eliminated by using the same container images, environment variables, and exact build steps to set up a reproduction.

This experience gave me a decent nudge towards getting rid of long-lived Vagrant instances for development and coalescing practice towards more consistent (but not identical!) environments for development, test and hosting.