Skip to content

(IAC-1595) MySQL maintenance #1472

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 24, 2022
Merged

(IAC-1595) MySQL maintenance #1472

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f47c356
(IAC-1595) MySQL maintenance
LukasAud May 24, 2022
8d5e3ea
Small improvements
LukasAud May 24, 2022
895ad50
Small adjustments
LukasAud May 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ jobs:
if: ${{ github.repository_owner == 'puppetlabs' }}
run: |
if [ '${{ github.repository_owner }}' == 'puppetlabs' ]; then
buildevents cmd $TRACE_ID $STEP_ID matrix_from_metadata -- bundle exec matrix_from_metadata_v2 --exclude-platforms '["Debian-9", "Debian-10", "Debian-11"]'
echo "::set-output name=matrix::{'platforms':[{'label':'AlmaLinux-8','provider':'provision::docker','image':'litmusimage/almalinux:8'},{'label':'CentOS-7','provider':'provision::docker','image':'litmusimage/centos:7'},{'label':'CentOS-8','provider':'provision::docker','image':'litmusimage/centos:stream8'},{'label':'Debian-10','provider':'provision::provision_service','image':'debian-10'},{'label':'Debian-11','provider':'provision::provision_service','image':'debian-11'},{'label':'Debian-9','provider':'provision::provision_service','image':'debian-9'},{'label':'OracleLinux-6','provider':'provision::docker','image':'litmusimage/oraclelinux:6'},{'label':'OracleLinux-7','provider':'provision::docker','image':'litmusimage/oraclelinux:7'},{'label':'RedHat-7','provider':'provision::provision_service','image':'rhel-7'},{'label':'RedHat-8','provider':'provision::provision_service','image':'rhel-8'},{'label':'Rocky-8','provider':'provision::docker','image':'litmusimage/rockylinux:8'},{'label':'SLES-12','provider':'provision::provision_service','image':'sles-12'},{'label':'SLES-15','provider':'provision::provision_service','image':'sles-15'},{'label':'Scientific-6','provider':'provision::docker','image':'litmusimage/scientificlinux:6'},{'label':'Scientific-7','provider':'provision::docker','image':'litmusimage/scientificlinux:7'},{'label':'Ubuntu-18.04','provider':'provision::docker','image':'litmusimage/ubuntu:18.04'},{'label':'Ubuntu-20.04','provider':'provision::provision_service','image':'ubuntu-2004-lts'}],'collection':['puppet6-nightly','puppet7-nightly']}"
else
echo "::set-output name=matrix::{}"
fi
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ jobs:
id: get-matrix
run: |
if [ '${{ github.repository_owner }}' == 'puppetlabs' ]; then
buildevents cmd $TRACE_ID $STEP_ID matrix_from_metadata -- bundle exec matrix_from_metadata_v2 --exclude-platforms '["Debian-9", "Debian-10", "Debian-11"]'
echo "::set-output name=matrix::{'platforms':[{'label':'AlmaLinux-8','provider':'provision::docker','image':'litmusimage/almalinux:8'},{'label':'CentOS-7','provider':'provision::docker','image':'litmusimage/centos:7'},{'label':'CentOS-8','provider':'provision::docker','image':'litmusimage/centos:stream8'},{'label':'Debian-10','provider':'provision::provision_service','image':'debian-10'},{'label':'Debian-11','provider':'provision::provision_service','image':'debian-11'},{'label':'Debian-9','provider':'provision::provision_service','image':'debian-9'},{'label':'OracleLinux-6','provider':'provision::docker','image':'litmusimage/oraclelinux:6'},{'label':'OracleLinux-7','provider':'provision::docker','image':'litmusimage/oraclelinux:7'},{'label':'RedHat-7','provider':'provision::provision_service','image':'rhel-7'},{'label':'RedHat-8','provider':'provision::provision_service','image':'rhel-8'},{'label':'Rocky-8','provider':'provision::docker','image':'litmusimage/rockylinux:8'},{'label':'SLES-12','provider':'provision::provision_service','image':'sles-12'},{'label':'SLES-15','provider':'provision::provision_service','image':'sles-15'},{'label':'Scientific-6','provider':'provision::docker','image':'litmusimage/scientificlinux:6'},{'label':'Scientific-7','provider':'provision::docker','image':'litmusimage/scientificlinux:7'},{'label':'Ubuntu-18.04','provider':'provision::docker','image':'litmusimage/ubuntu:18.04'},{'label':'Ubuntu-20.04','provider':'provision::provision_service','image':'ubuntu-2004-lts'}],'collection':['puppet6','puppet7']}"
else
echo "::set-output name=matrix::{}"
fi
Expand Down
12 changes: 9 additions & 3 deletions lib/puppet/provider/mysql_grant/mysql.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,15 +68,21 @@ def self.instances
'UPDATE']
sorted_privileges = ['ALL']

# Currently there is an issue with the behaviour of the module which was highlighted by the 'complex test' test case in 'mysql_grant_spec'. The module, upon retrieving all privileges from an
# user, does not take into account that the latest version of mysql now includes dynamic privileges which are returned alongside the original static privileges and are set by 'ALL PRIVILEGES'
# (shortened to 'ALL'). This is a workaround to remove the unnecesary privileges from the sorted_privileges list which is used to check for idempotency in test cases.
# The following two elsif blocks of code are a workaround for issue #1474.
elsif sorted_privileges == ['ALL', 'APPLICATION_PASSWORD_ADMIN', 'AUDIT_ABORT_EXEMPT', 'AUDIT_ADMIN', 'AUTHENTICATION_POLICY_ADMIN', 'BACKUP_ADMIN', 'BINLOG_ADMIN', 'BINLOG_ENCRYPTION_ADMIN',
'CLONE_ADMIN', 'CONNECTION_ADMIN', 'ENCRYPTION_KEY_ADMIN', 'FLUSH_OPTIMIZER_COSTS', 'FLUSH_STATUS', 'FLUSH_TABLES', 'FLUSH_USER_RESOURCES',
'GROUP_REPLICATION_ADMIN', 'GROUP_REPLICATION_STREAM', 'INNODB_REDO_LOG_ARCHIVE', 'INNODB_REDO_LOG_ENABLE', 'PASSWORDLESS_USER_ADMIN', 'PERSIST_RO_VARIABLES_ADMIN',
'REPLICATION_APPLIER', 'REPLICATION_SLAVE_ADMIN', 'RESOURCE_GROUP_ADMIN', 'RESOURCE_GROUP_USER', 'ROLE_ADMIN', 'SERVICE_CONNECTION_ADMIN',
'SESSION_VARIABLES_ADMIN', 'SET_USER_ID', 'SHOW_ROUTINE', 'SYSTEM_USER', 'SYSTEM_VARIABLES_ADMIN', 'TABLE_ENCRYPTION_ADMIN', 'XA_RECOVER_ADMIN']
sorted_privileges = ['ALL']

elsif sorted_privileges == ['ALL', 'APPLICATION_PASSWORD_ADMIN', 'AUDIT_ABORT_EXEMPT', 'AUDIT_ADMIN', 'AUTHENTICATION_POLICY_ADMIN', 'BACKUP_ADMIN', 'BINLOG_ADMIN', 'BINLOG_ENCRYPTION_ADMIN',
'CLONE_ADMIN', 'CONNECTION_ADMIN', 'ENCRYPTION_KEY_ADMIN', 'FLUSH_OPTIMIZER_COSTS', 'FLUSH_STATUS', 'FLUSH_TABLES', 'FLUSH_USER_RESOURCES',
'GROUP_REPLICATION_ADMIN', 'GROUP_REPLICATION_STREAM', 'INNODB_REDO_LOG_ARCHIVE', 'INNODB_REDO_LOG_ENABLE', 'PASSWORDLESS_USER_ADMIN', 'PERSIST_RO_VARIABLES_ADMIN',
'REPLICATION_APPLIER', 'REPLICATION_SLAVE_ADMIN', 'RESOURCE_GROUP_ADMIN', 'RESOURCE_GROUP_USER', 'ROLE_ADMIN', 'SENSITIVE_VARIABLES_OBSERVER',
'SERVICE_CONNECTION_ADMIN', 'SESSION_VARIABLES_ADMIN', 'SET_USER_ID', 'SHOW_ROUTINE', 'SYSTEM_USER', 'SYSTEM_VARIABLES_ADMIN', 'TABLE_ENCRYPTION_ADMIN',
'XA_RECOVER_ADMIN']
sorted_privileges = ['ALL']
end

instance_configs[name] = {
Expand Down
23 changes: 17 additions & 6 deletions manifests/backup/xtrabackup.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,23 @@
require => Class['mysql::server::root_password'],
}

mysql_grant { "${backupuser}@localhost/*.*":
ensure => $ensure,
user => "${backupuser}@localhost",
table => '*.*',
privileges => ['RELOAD', 'PROCESS', 'LOCK TABLES', 'REPLICATION CLIENT'],
require => Mysql_user["${backupuser}@localhost"],
if $::osfamily == 'debian' and $::operatingsystemmajrelease == '11' {
mysql_grant { "${backupuser}@localhost/*.*":
ensure => $ensure,
user => "${backupuser}@localhost",
table => '*.*',
privileges => ['BINLOG MONITOR', 'RELOAD', 'PROCESS', 'LOCK TABLES'],
require => Mysql_user["${backupuser}@localhost"],
}
}
else {
mysql_grant { "${backupuser}@localhost/*.*":
ensure => $ensure,
user => "${backupuser}@localhost",
table => '*.*',
privileges => ['RELOAD', 'PROCESS', 'LOCK TABLES', 'REPLICATION CLIENT'],
require => Mysql_user["${backupuser}@localhost"],
}
}
}

Expand Down
1 change: 1 addition & 0 deletions manifests/params.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@
$ruby_package_name = $facts['os']['release']['major'] ? {
'9' => 'ruby-mysql2', # stretch
'10' => 'ruby-mysql2', # buster
'11' => 'ruby-mysql2', # bullseye
'16.04' => 'ruby-mysql', # xenial
'18.04' => 'ruby-mysql2', # bionic
'20.04' => 'ruby-mysql2', # focal
Expand Down
1 change: 0 additions & 1 deletion manifests/server.pp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,6 @@
~> Class['mysql::server::service']
}


if $_options['mysqld']['ssl-disable'] {
notify { 'ssl-disable':
message => 'Disabling SSL is evil! You should never ever do this except
Expand Down
6 changes: 5 additions & 1 deletion spec/acceptance/00_mysql_server_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,11 @@ class { '::mysql::server':

it 'can be set' do
apply_manifest(pp, catch_failures: true) do |r|
expect(r.stderr).to be_empty
if (os[:family] == 'debian' && os[:release].to_i > 9) && Gem::Version.new(run_shell('puppet --version').stdout) < Gem::Version.new('7.0.0')
expect(r.stderr).to match(%r{locale environment variables were bad; continuing with LANG=C LC_ALL=C})
else
expect(r.stderr).to be_empty
end
end
end
end
Expand Down
14 changes: 5 additions & 9 deletions spec/acceptance/04_mysql_backup_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ class { 'mysql::server::backup':
end

context 'with xtrabackup enabled' do
context 'should work with no errors', if: ((os[:family] == 'debian') || (os[:family] == 'ubuntu' && os[:release] =~ %r{^16\.04|^18\.04}) || (os[:family] == 'redhat' && os[:release].to_i > 7)) do
context 'should work with no errors', if: ((os[:family] == 'debian') || (os[:family] == 'ubuntu') || (os[:family] == 'redhat' && os[:release].to_i > 7)) do
pp = <<-MANIFEST
class { 'mysql::server': root_password => 'password' }
mysql::db { [
Expand All @@ -142,11 +142,7 @@ class { 'mysql::server': root_password => 'password' }
}
case $facts['os']['family'] {
/Debian/: {
if versioncmp($::operatingsystemmajrelease, '8') >= 0 {
$source_url = "http://repo.percona.com/apt/percona-release_1.0-22.generic_all.deb"
} else {
$source_url = "http://repo.percona.com/apt/percona-release_latest.${facts['os']['distro']['codename']}_all.deb"
}
$source_url = "http://repo.percona.com/apt/percona-release_latest.${facts['os']['distro']['codename']}_all.deb"

file { '/tmp/percona-release_latest.deb':
ensure => present,
Expand Down Expand Up @@ -209,7 +205,7 @@ class { 'mysql::server::backup':
end
end

describe 'xtrabackup.sh', if: Gem::Version.new(mysql_version) < Gem::Version.new('5.7.0') && ((os[:family] == 'debian' && os[:release].to_i >= 9) || (os[:family] == 'ubuntu' && os[:release] =~ %r{^16\.04|^18\.04}) || (os[:family] == 'redhat' && os[:release].to_i > 7)) do # rubocop:disable Layout/LineLength
describe 'xtrabackup.sh', if: Gem::Version.new(mysql_version) < Gem::Version.new('5.7.0') && ((os[:family] == 'debian' && os[:release].to_i >= 9) || (os[:family] == 'ubuntu') || (os[:family] == 'redhat' && os[:release].to_i > 7)) do # rubocop:disable Layout/LineLength
it 'runs xtrabackup.sh full backup with no errors' do
run_shell('/usr/local/sbin/xtrabackup.sh --target-dir=/tmp/xtrabackups/$(date +%F)_full --backup 2>&1 | tee /tmp/xtrabackup_full.log') do |r|
expect(r.exit_code).to be_zero
Expand Down Expand Up @@ -254,7 +250,7 @@ class { 'mysql::server::backup':
end

context 'with xtrabackup enabled and incremental backups disabled' do
context 'should work with no errors', if: ((os[:family] == 'debian' && os[:release].to_i >= 9) || (os[:family] == 'ubuntu' && os[:release] =~ %r{^16\.04|^18\.04}) || (os[:family] == 'redhat' && os[:release].to_i > 7)) do # rubocop:disable Layout/LineLength
context 'should work with no errors', if: ((os[:family] == 'debian') || (os[:family] == 'ubuntu') || (os[:family] == 'redhat' && os[:release].to_i > 7)) do
pp = <<-MANIFEST
class { 'mysql::server': root_password => 'password' }
mysql::db { [
Expand Down Expand Up @@ -323,7 +319,7 @@ class { 'mysql::server::backup':
end
end

describe 'xtrabackup.sh', if: Gem::Version.new(mysql_version) < Gem::Version.new('5.7.0') && ((os[:family] == 'debian' && os[:release].to_i >= 9) || (os[:family] == 'ubuntu' && os[:release] =~ %r{^16\.04|^18\.04}) || (os[:family] == 'redhat' && os[:release].to_i > 7)) do # rubocop:disable Layout/LineLength
describe 'xtrabackup.sh', if: Gem::Version.new(mysql_version) < Gem::Version.new('5.7.0') && ((os[:family] == 'debian' && os[:release].to_i >= 9) || (os[:family] == 'ubuntu') || (os[:family] == 'redhat' && os[:release].to_i > 7)) do # rubocop:disable Layout/LineLength
it 'runs xtrabackup.sh with no errors' do
run_shell('/usr/local/sbin/xtrabackup.sh --target-dir=/tmp/xtrabackups/$(date +%F_%H-%M-%S) --backup 2>&1 | tee /tmp/xtrabackup.log') do |r|
expect(r.exit_code).to be_zero
Expand Down
4 changes: 2 additions & 2 deletions spec/acceptance/types/mysql_login_path_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
support_bin_dir = '/root/mysql_login_path'
if os[:family] == 'redhat' && os[:release].to_i == 8
mysql_version = '8.0'
elsif os[:family] == 'debian' && os[:release] =~ %r{9|10}
elsif os[:family] == 'debian' && os[:release] =~ %r{9|10|11}
mysql_version = '8.0'
elsif os[:family] == 'ubuntu' && os[:release] =~ %r{16\.04|18\.04}
elsif os[:family] == 'ubuntu' && os[:release] =~ %r{18\.04|20\.04}
mysql_version = '5.7'
end

Expand Down
7 changes: 6 additions & 1 deletion spec/classes/mysql_backup_xtrabackup_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,12 @@ class { 'mysql::server': }
ensure: 'present',
user: 'backupuser@localhost',
table: '*.*',
privileges: ['RELOAD', 'PROCESS', 'LOCK TABLES', 'REPLICATION CLIENT'],
privileges:
if facts[:osfamily] == 'Debian' && Puppet::Util::Package.versioncmp(facts[:operatingsystemmajrelease], '11') == 0
['BINLOG MONITOR', 'RELOAD', 'PROCESS', 'LOCK TABLES']
else
['RELOAD', 'PROCESS', 'LOCK TABLES', 'REPLICATION CLIENT']
end,
)
.that_requires('Mysql_user[backupuser@localhost]')
end
Expand Down
4 changes: 2 additions & 2 deletions spec/spec_helper_acceptance_local.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ def export_locales
LitmusHelper.instance.run_shell('echo export LANGUAGE=en_US.UTF-8 >> /etc/profile.d/my-custom.lang.sh')
LitmusHelper.instance.run_shell('echo export LC_COLLATE=C >> /etc/profile.d/my-custom.lang.sh')
LitmusHelper.instance.run_shell('echo export LC_CTYPE=en_US.UTF-8 >> /etc/profile.d/my-custom.lang.sh')
LitmusHelper.instance.run_shell('source /etc/profile.d/my-custom.lang.sh')
LitmusHelper.instance.run_shell('. /etc/profile.d/my-custom.lang.sh')
LitmusHelper.instance.run_shell('echo export LC_ALL="C" >> ~/.bashrc')
LitmusHelper.instance.run_shell('source ~/.bashrc')
LitmusHelper.instance.run_shell('. ~/.bashrc')
end

def fetch_charset
Expand Down