Как восстановить Sentry после неудачного обновления23.07.2020 08:02

?v=1

Всем привет. Я хочу рассказать о том, как проходило восстановление Sentry после неудачного обновления.
Что же такое Sentry?

Это система полного отслеживания ошибок с открытым исходным кодом, которая поддерживает широкий спектр серверных, браузерных, настольных и родных мобильных языков и сред, включая PHP, Node. js, Python, Ruby, C #, Java, Go, React, Angular, Vue, JavaScript и другие.

Немного о том, как просто и непринуждённо всё поломалось.

Жили мы на версии 9.0.0 и пришло время обновиться. Пощупав web интерфейс 10.0.0, принял решение обновиться на неё. Это была самая большая ошибка!

Обновление прошло в штатном режиме. Сначала переход на 9.1.2 затем, на 10.0.0 (иначе нельзя). Далее обнаружил, что не запускается worker. Причиной этому добавление новых приложений и зависимостей.

Попытки собрать приложения и засунуть их в k8s не увенчались успехом.

Было принято решение откатиться назад. Но тут меня ждало разочарование. После миграции на 10 версию, часть данных переезжает в clickhouse и удаляется из Postgres.

После 3 часов работ и привлечения DBA эксперта данные были восстановлены до рабочего состояния.

Как?

Сделали рядом инстанс PG, закинули туда схему от версии 9.0.0 и начали потабличное восстановление. Конечно, без ошибок и напильника тут не обошлось.

На этом история не заканчивается. Пришло время всё же обновиться на версию 9.1.2.

Собрал образ, запускаю: 

sentry upgrade


и видим вот такое чудесное сообщение: 

 ! I'm not trusting myself; either fix this yourself by fiddling
 ! with the south_migrationhistory table, or pass --delete-ghost-migrations
 ! to South to have it delete ALL of these records (this may not be good).
Exception in thread Thread-1 (most likely raised during interpreter shutdown)


После долгих поисков, проб и ошибок было найдено решение.

Необходимо восстановить схему данных текущей версии.

Для этого клонируем репозиторий: 

git clone https://github.com/getsentry/onpremise.git


Переходим в ветку 9.1.2, так как ветки 9.0.0 уже нет.

cd onpremise && git checkout tags/9.1.2


Далее правим образ в Dockerfile: 

#cat Dockerfile 
ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.0.0}-onbuild

Затем правим compose file

docker-compose.yml 
# NOTE: This docker-compose.yml is meant to be just an example of how
# you could accomplish this on your own. It is not intended to work in
# all use-cases and must be adapted to fit your needs. This is merely
# a guideline.

# See docs.getsentry.com/on-premise/server/ for full
# instructions

version: '3.4'

x-defaults: &defaults
  restart: unless-stopped
  build:
    context: .
  depends_on:
    - redis
   # - postgres
    - memcached
    - smtp
  env_file: .env
  environment:
    SENTRY_MEMCACHED_HOST: memcached
    SENTRY_REDIS_HOST: redis
    SENTRY_POSTGRES_HOST: 'sentry.cl.ats'
    SENTRY_DB_USER: 'sentryDB'
    SENTRY_DB_NAME: 'sentry_user'
    SENTRY_DB_PASSWORD: 'sentry_passwd'
    SENTRY_POSTGRES_PORT: 5432
    SENTRY_EMAIL_HOST: smtp
  volumes:
    - sentry-data:/var/lib/sentry/files


services:
  smtp:
    restart: unless-stopped
    image: tianon/exim4

  memcached:
    restart: unless-stopped
    image: memcached:1.5-alpine

  redis:
    restart: unless-stopped
    image: redis:3.2-alpine

  #postgres:
  #  restart: unless-stopped
  #  image: postgres:9.5
  #  environment:
  #    POSTGRES_HOST_AUTH_METHOD: 'trust'
  #  volumes:
  #    - sentry-postgres:/var/lib/postgresql/data

  web:
    <<: *defaults
    ports:
      - '9000:9000'

  cron:
    <<: *defaults
    command: run cron

  worker:
    <<: *defaults
    command: run worker


volumes:
    sentry-data:
      external: true
    #sentry-postgres:
    #  external: true


Немного пояснений. Так как у меня есть отдельный инстанс базы данных, то создание через compose пропускаю, также не нужен volume sentry-postgres.

После данных манипуляций запускаем сборку: 

./install.sh


В процессе может выскочить ошибка, о которой я упомянул выше. Не обращаем внимания и после того, как процесс закончится, запускаем следующую команду: 

docker-compose run --rm web sentry django migrate --delete-ghost-migrations


Примерный вывод 
Creating network "onpremise_default" with the default driver
Creating onpremise_memcached_1 ... done
Creating onpremise_smtp_1      ... done
Creating onpremise_redis_1     ... done
21:30:26 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for sentry:
- Nothing to migrate.
 - Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
 - Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
 - Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
 - Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Running migrations for tagstore:
- Nothing to migrate.
 - Loading initial data for tagstore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jira_ac:
- Nothing to migrate.
 - Loading initial data for jira_ac.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hipchat_ac:
- Nothing to migrate.
 - Loading initial data for hipchat_ac.
Installed 0 object(s) from 0 fixture(s)


Далее можно запускать upgrade.

docker-compose run --rm web sentry upgrade


Stdout 
Starting onpremise_smtp_1      ... done
Starting onpremise_memcached_1 ... done
Starting onpremise_redis_1     ... done
21:30:48 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

Synced:
 > django.contrib.admin
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.messages
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.staticfiles
 > crispy_forms
 > debug_toolbar
 > raven.contrib.django.raven_compat
 > rest_framework
 > sentry.plugins.sentry_interface_types
 > sentry.plugins.sentry_mail
 > sentry.plugins.sentry_urls
 > sentry.plugins.sentry_useragents
 > sentry.plugins.sentry_webhooks
 > sudo
 > south
 > sentry_plugins.slack

Not synced (use migrations):
 - sentry
 - sentry.nodestore
 - sentry.search
 - social_auth
 - sentry.tagstore
 - sentry_plugins.jira_ac
 - sentry_plugins.hipchat_ac
(use ./manage.py migrate to migrate these)
Running migrations for sentry:
- Nothing to migrate.
 - Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
 - Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
 - Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
 - Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Running migrations for tagstore:
- Nothing to migrate.
 - Loading initial data for tagstore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jira_ac:
- Nothing to migrate.
 - Loading initial data for jira_ac.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hipchat_ac:
- Nothing to migrate.
 - Loading initial data for hipchat_ac.
Installed 0 object(s) from 0 fixture(s)
Creating missing DSNs
Correcting Group.num_comments counter


Итак, схема восстановлена. Можно обновиться на 9.1.2.

Для этого в Dockerfile меняем версию на 9.1.2: 

ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild

И запускаем install.sh

./install.sh


Если всё сделано верно, то процесс завершится без ошибок. Вывод должен быть примерно такой:

Stdout 
# ./install.sh 
Checking minimum requirements...

Creating volumes for persistent storage...
Created sentry-data.

.env already exists, skipped creation.

Building and tagging Docker images...

smtp uses an image, skipping
memcached uses an image, skipping
redis uses an image, skipping
Building web
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Running in 6c97f9fcaf63
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Removing intermediate container 6c97f9fcaf63
 ---> Running in 9e0f65ee3af6
Removing intermediate container 9e0f65ee3af6
 ---> Running in 09754c44319c
Removing intermediate container 09754c44319c
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_web:latest
Building cron
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_cron:latest
Building worker
Step 1/2 : ARG SENTRY_IMAGE
Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
# Executing 4 build triggers
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> Using cache
 ---> a12fa31c2675

Successfully built a12fa31c2675
Successfully tagged onpremise_worker:latest

Docker images built.

Generating secret key...
Secret key written to .env

Setting up database...
Starting onpremise_smtp_1      ... done
Starting onpremise_redis_1     ... done
Starting onpremise_memcached_1 ... done
21:35:07 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
21:35:10 [INFO] sentry.plugins.github: apps-not-configured
Syncing...
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
Migrating...
Running migrations for sentry:
 - Migrating forwards to 0472_auto__add_field_sentryapp_author.
 > sentry:0424_auto__add_field_integration_status
 > sentry:0425_auto__add_index_pullrequest_organization_id_merge_commit_sha
 > sentry:0425_remove_invalid_github_idps
 > sentry:0426_auto__add_sentryappinstallation__add_sentryapp__add_field_user_is_sent
 > sentry:0427_auto__add_eventattachment__add_unique_eventattachment_project_id_event
 > sentry:0428_auto__add_index_eventattachment_project_id_date_added
 > sentry:0429_auto__add_integrationexternalproject__add_unique_integrationexternalpr
 > sentry:0430_auto__add_field_organizationintegration_status
 > sentry:0431_auto__add_field_externalissue_metadata
 > sentry:0432_auto__add_field_relay_is_internal
 > sentry:0432_auto__add_index_userreport_date_added__add_index_eventattachment_date_
 > sentry:0433_auto__add_field_relay_is_internal__add_field_userip_country_code__add_
 > sentry:0434_auto__add_discoversavedqueryproject__add_unique_discoversavedqueryproj
 > sentry:0435_auto__add_field_discoversavedquery_created_by
 > sentry:0436_rename_projectdsymfile_to_projectdebugfile
 > sentry:0437_auto__add_field_sentryapp_status
 > sentry:0438_auto__add_index_sentryapp_status__chg_field_sentryapp_proxy_user__chg_
 > sentry:0439_auto__chg_field_sentryapp_owner
 > sentry:0440_auto__del_unique_projectdebugfile_project_debug_id__add_index_projectd
 > sentry:0441_auto__add_field_projectdebugfile_data
 > sentry:0442_auto__add_projectcficachefile__add_unique_projectcficachefile_project_
 > sentry:0443_auto__add_field_organizationmember_token_expires_at
 > sentry:0443_auto__del_dsymapp__del_unique_dsymapp_project_platform_app_id__del_ver
 > sentry:0444_auto__add_sentryappavatar__add_field_sentryapp_redirect_url__add_field
 > sentry:0445_auto__add_promptsactivity__add_unique_promptsactivity_user_feature_org
 > sentry:0446_auto__add_index_promptsactivity_project_id
 > sentry:0447_auto__del_field_promptsactivity_organization__add_field_promptsactivit
 > sentry:0448_auto__add_field_sentryapp_is_alertable
 > sentry:0449_auto__chg_field_release_owner
 > sentry:0450_auto__del_grouphashtombstone__del_unique_grouphashtombstone_project_ha
 > sentry:0451_auto__del_field_projectbookmark_project_id__add_field_projectbookmark_
 > sentry:0452_auto__add_field_sentryapp_events
 > sentry:0452_auto__del_field_releaseenvironment_organization_id__del_field_releasee
 > sentry:0453_auto__add_index_releasefile_release_name
 > sentry:0454_resolve_duplicate_0452
 > sentry:0455_auto__add_field_groupenvironment_first_seen
 > sentry:0456_auto__add_dashboard__add_unique_dashboard_organization_title__add_widg
 > sentry:0457_auto__add_field_savedsearch_is_global__chg_field_savedsearch_project__
 > sentry:0457_auto__add_monitorcheckin__add_monitor__add_index_monitor_type_next_che
 > sentry:0458_global_searches_data_migration
Saved Searchs: 100% |########################################################################################################################################################################| Time: 0:00:07
 > sentry:0459_global_searches_unique_constraint
 > sentry:0460_auto__add_field_servicehook_organization_id
 > sentry:0461_event_attachment_indexes
 > sentry:0462_auto__add_servicehookproject
 > sentry:0462_releaseenvironment_project_id
 > sentry:0463_backfill_service_hook_project
 > sentry:0464_auto__add_sentryappcomponent__add_field_sentryapp_schema
 > sentry:0464_groupenvironment_foreignkeys
 > sentry:0465_sync
 > sentry:0466_auto__add_platformexternalissue__add_unique_platformexternalissue_grou
 > sentry:0467_backfill_integration_status
 > sentry:0468_auto__add_field_projectdebugfile_code_id__add_index_projectdebugfile_p
 > sentry:0468_recent_search
 > sentry:0469_fix_state
 > sentry:0470_org_saved_search
 > sentry:0471_global_saved_search_types
 > sentry:0472_auto__add_field_sentryapp_author
The following content types are stale and need to be deleted:

    sentry | dsymobject
    sentry | dsymapp
    sentry | useridentity
    sentry | dsymsymbol
    sentry | dsymsdk
    sentry | globaldsymfile
    sentry | versiondsymfile
    sentry | projectdsymfile
    sentry | grouphashtombstone
    sentry | minidumpfile
    sentry | dsymbundle

Any objects related to these content types by a foreign key will also
be deleted. Are you sure you want to delete these content types?
If you're unsure, answer 'no'.

    Type 'yes' to continue, or 'no' to cancel: yes
Running migrations for sentry.nodestore:
- Nothing to migrate.
Running migrations for sentry.search:
- Nothing to migrate.
Running migrations for social_auth:
- Nothing to migrate.
Running migrations for sentry.tagstore:
- Nothing to migrate.
Running migrations for sentry_plugins.hipchat_ac:
- Nothing to migrate.
Running migrations for sentry_plugins.jira_ac:
- Nothing to migrate.

Synced:
 > django.contrib.admin
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.messages
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.staticfiles
 > crispy_forms
 > debug_toolbar
 > rest_framework
 > sentry.plugins.sentry_interface_types
 > sentry.plugins.sentry_mail
 > sentry.plugins.sentry_urls
 > sentry.plugins.sentry_useragents
 > sentry.plugins.sentry_webhooks
 > sudo
 > south

Migrated:
 - sentry
 - sentry.nodestore
 - sentry.search
 - social_auth
 - sentry.tagstore
 - sentry_plugins.hipchat_ac
 - sentry_plugins.jira_ac
Creating missing DSNs
Correcting Group.num_comments counter
Cleaning up...

----------------
You're all done! Run the following command get Sentry running:

  docker-compose up -d


Всё, версия 9.1.2 установлена.

Выводы.

Это всем известные прописные истины. Не спешите, делайте резервные копии и проверяйте всё несколько раз.

Благодарю за внимание!

© Habrahabr.ru