Merge branch 'main' into ignore_mysqld_user

19 files changed, 172 insertions, 56 deletions

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 0a14472..b678f57 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -12,7 +12,7 @@ jobs:
         python-version: [3.6, 3.7, 3.8, 3.9]
         include:
           - python-version: 3.6
-            os: ubuntu-16.04 # MySQL 5.7.32
+            os: ubuntu-18.04 # MySQL 5.7.32
           - python-version: 3.7
             os: ubuntu-18.04 # MySQL 5.7.32
           - python-version: 3.8
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 124b19a..cac4f04 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -49,16 +49,16 @@ You'll always get credit for your work.
     $ pip install --editable .
     ```
 
-6. Create a branch for your bugfix or feature based off the `master` branch:
+6. Create a branch for your bugfix or feature based off the `main` branch:
 
     ```bash
-    $ git checkout -b <name-of-bugfix-or-feature> master
+    $ git checkout -b <name-of-bugfix-or-feature> main
     ```
 
 7. While you work on your bugfix or feature, be sure to pull the latest changes from `upstream`. This ensures that your local codebase is up-to-date:
 
     ```bash
-    $ git pull upstream master
+    $ git pull upstream main
     ```
 
 8. When your work is ready for the mycli team to review it, push your branch to your fork:
@@ -95,7 +95,7 @@ credentials to use by setting the applicable environment variables:
 
 ```bash
 $ export PYTEST_HOST=localhost
-$ export PYTEST_USER=user
+$ export PYTEST_USER=mycli
 $ export PYTEST_PASSWORD=myclirocks
 $ export PYTEST_PORT=3306
 $ export PYTEST_CHARSET=utf8
@@ -104,6 +104,14 @@ $ export PYTEST_CHARSET=utf8
 The default values are `localhost`, `root`, no password, `3306`, and `utf8`.
 You only need to set the values that differ from the defaults.
 
+If you would like to run the tests as a user with only the necessary privileges,
+create a `mycli` user and run the following grant statements.
+
+```sql
+GRANT ALL PRIVILEGES ON `mycli_%`.* TO 'mycli'@'localhost';
+GRANT SELECT ON mysql.* TO 'mycli'@'localhost';
+GRANT SELECT ON performance_schema.* TO 'mycli'@'localhost';
+```
 
 ### CLI Tests
 
@@ -135,3 +143,25 @@ $ ./setup.py lint --fix
 ```
 
 Be sure to commit and push any PEP 8 fixes.
+
+## Releasing a new version of mycli
+
+You have been made the maintainer of `mycli`? Congratulations! We have a release script to help you:
+
+```sh
+> python release.py --help
+Usage: release.py [options]
+
+Options:
+  -h, --help           show this help message and exit
+  -c, --confirm-steps  Confirm every step. If the step is not confirmed, it
+                       will be skipped.
+  -d, --dry-run        Print out, but not actually run any steps.
+```
+
+To release a new version of the package:
+
+* Create and merge a PR to bump the version in the changelog ([example PR](https://github.com/dbcli/mycli/pull/1043)).
+* Pull `main` and bump the version number inside `mycli/__init__.py`. Do not check in - the release script will do that.
+* Make sure you have the dev requirements installed: `pip install -r requirements-dev.txt -U --upgrade-strategy only-if-needed`.
+* Finally, run the release script: `python release.py`.
diff --git a/README.md b/README.md
index cc04a91..e15f505 100644
--- a/README.md
+++ b/README.md
@@ -141,7 +141,7 @@ If you're interested in contributing to this project, first of all I would like
 to extend my heartfelt gratitude. I've written a small doc to describe how to
 get this running in a development setup.
 
-https://github.com/dbcli/mycli/blob/master/CONTRIBUTING.md
+https://github.com/dbcli/mycli/blob/main/CONTRIBUTING.md
 
 Please feel free to reach out to me if you need help.
 
@@ -151,6 +151,22 @@ Twitter: [@amjithr](http://twitter.com/amjithr)
 
 ## Detailed Install Instructions:
 
+### Arch, Manjaro
+
+You can install the mycli package available in the AUR:
+
+```
+$ yay -S mycli
+```
+
+### Debian, Ubuntu
+
+On Debian, Ubuntu distributions, you can easily install the mycli package using apt:
+
+```
+$ sudo apt-get install mycli
+```
+
 ### Fedora
 
 Fedora has a package available for mycli, install it using dnf:
@@ -164,13 +180,13 @@ $ sudo dnf install mycli
 I haven't built an RPM package for mycli for RHEL or Centos yet. So please use `pip` to install `mycli`. You can install pip on your system using:
 
 ```
-$ sudo yum install python-pip
+$ sudo yum install python3-pip
 ```
 
 Once that is installed, you can install mycli as follows:
 
 ```
-$ sudo pip install mycli
+$ sudo pip3 install mycli
 ```
 
 ### Windows
diff --git a/changelog.md b/changelog.md
index ff1e6e5..159299d 100644
--- a/changelog.md
+++ b/changelog.md
@@ -1,11 +1,41 @@
+
 TBD
 ===
 
+Internal:
+---------
+* Pin `cryptography` to suppress `paramiko` warning, helping CI complete and presumably affecting some users.
+* Upgrade some dev requirements
+* Change tests to always use databases prefixed with 'mycli_' for better security
+
 Bug Fixes:
 ----------
+* Support for some MySQL compatible databases, which may not implement connection_id().
+* Fix the status command to work with missing 'Flush_commands' (mariadb)
 * Ignore the user of the system [myslqd] config.
 
 
+1.25.0 (2022/04/02)
+===================
+
+Features:
+---------
+* Add `beep_after_seconds` option to `~/.myclirc`, to ring the terminal bell after long queries.
+
+
+1.24.4 (2022/03/30)
+===================
+
+Internal:
+---------
+* Upgrade Ubuntu VM for runners as Github has deprecated it
+
+Bug Fixes:
+----------
+* Change in main.py - Replace the `click.get_terminal_size()` with `shutil.get_terminal_size()`
+
+
+
 1.24.3 (2022/01/20)
 ===================
 
diff --git a/mycli/AUTHORS b/mycli/AUTHORS
index 232afd4..328805d 100644
--- a/mycli/AUTHORS
+++ b/mycli/AUTHORS
@@ -39,6 +39,7 @@ Contributors:
   * Georgy Frolov
   * Heath Naylor
   * Huachao Mao
+  * Ishaan Bhimwal
   * Jakub Boukal
   * jbruno
   * Jerome Provensal
@@ -82,12 +83,15 @@ Contributors:
   * xeron
   * Yang Zou
   * Yasuhiro Matsumoto
+  * Yuanchun Shang
   * Zach DeCook
   * Zane C. Bowers-Hadley
   * zer09
   * Zhaolong Zhu
   * Zhidong
   * Zhongyang Guan
+  * Arvind Mishra
+  * Kevin Schmeichel
 
 Created by:
 -----------
diff --git a/mycli/__init__.py b/mycli/__init__.py
index f7704b3..8de33c0 100644
--- a/mycli/__init__.py
+++ b/mycli/__init__.py
@@ -1 +1 @@
-__version__ = '1.24.3'
+__version__ = '1.25.0'
diff --git a/mycli/completion_refresher.py b/mycli/completion_refresher.py
index 124068a..8eb3de9 100644
--- a/mycli/completion_refresher.py
+++ b/mycli/completion_refresher.py
@@ -47,7 +47,7 @@ class CompletionRefresher(object):
     def _bg_refresh(self, sqlexecute, callbacks, completer_options):
         completer = SQLCompleter(**completer_options)
 
-        # Create a new pgexecute method to popoulate the completions.
+        # Create a new pgexecute method to populate the completions.
         e = sqlexecute
         executor = SQLExecute(e.dbname, e.user, e.password, e.host, e.port,
                               e.socket, e.charset, e.local_infile, e.ssl,
diff --git a/mycli/main.py b/mycli/main.py
index bd058a8..0561af8 100755
--- a/mycli/main.py
+++ b/mycli/main.py
@@ -2,6 +2,7 @@ from collections import defaultdict
 from io import open
 import os
 import sys
+import shutil
 import traceback
 import logging
 import threading
@@ -137,6 +138,7 @@ class MyCli(object):
         self.multi_line = c['main'].as_bool('multi_line')
         self.key_bindings = c['main']['key_bindings']
         special.set_timing_enabled(c['main'].as_bool('timing'))
+        self.beep_after_seconds = float(c['main']['beep_after_seconds'] or 0)
 
         FavoriteQueries.instance = FavoriteQueries.from_config(self.config)
 
@@ -446,7 +448,7 @@ class MyCli(object):
         if not any(v for v in ssl.values()):
             ssl = None
 
-        # if the passwd is not specfied try to set it using the password_file option
+        # if the passwd is not specified try to set it using the password_file option
         password_from_file = self.get_password_from_file(password_file)
         passwd = passwd or password_from_file
 
@@ -721,6 +723,8 @@ class MyCli(object):
                             self.output(formatted, status)
                         except KeyboardInterrupt:
                             pass
+                        if self.beep_after_seconds > 0 and t >= self.beep_after_seconds:
+                            self.echo('\a', err=True, nl=False)
                         if special.is_timing_enabled():
                             self.echo('Time: %0.03fs' % t)
                     except KeyboardInterrupt:
@@ -736,19 +740,23 @@ class MyCli(object):
             except KeyboardInterrupt:
                 # get last connection id
                 connection_id_to_kill = sqlexecute.connection_id
-                logger.debug("connection id to kill: %r", connection_id_to_kill)
-                # Restart connection to the database
-                sqlexecute.connect()
-                try:
-                    for title, cur, headers, status in sqlexecute.run('kill %s' % connection_id_to_kill):
-                        status_str = str(status).lower()
-                        if status_str.find('ok') > -1:
-                            logger.debug("cancelled query, connection id: %r, sql: %r",
-                                         connection_id_to_kill, text)
-                            self.echo("cancelled query", err=True, fg='red')
-                except Exception as e:
-                    self.echo('Encountered error while cancelling query: {}'.format(e),
-                              err=True, fg='red')
+                # some mysql compatible databases may not implemente connection_id()
+                if connection_id_to_kill > 0:
+                    logger.debug("connection id to kill: %r", connection_id_to_kill)
+                    # Restart connection to the database
+                    sqlexecute.connect()
+                    try:
+                        for title, cur, headers, status in sqlexecute.run('kill %s' % connection_id_to_kill):
+                            status_str = str(status).lower()
+                            if status_str.find('ok') > -1:
+                                logger.debug("cancelled query, connection id: %r, sql: %r",
+                                             connection_id_to_kill, text)
+                                self.echo("cancelled query", err=True, fg='red')
+                    except Exception as e:
+                        self.echo('Encountered error while cancelling query: {}'.format(e),
+                                  err=True, fg='red')
+                else:
+                  logger.debug("Did not get a connection id, skip cancelling query")
             except NotImplementedError:
                 self.echo('Not Yet Implemented.', fg="yellow")
             except OperationalError as e:
@@ -1055,7 +1063,7 @@ class MyCli(object):
         """Get the number of lines to reserve for the completion menu."""
         reserved_space_ratio = .45
         max_reserved_space = 8
-        _, height = click.get_terminal_size()
+        _, height = shutil.get_terminal_size()
         return min(int(round(height * reserved_space_ratio)), max_reserved_space)
 
     def get_last_query(self):
diff --git a/mycli/myclirc b/mycli/myclirc
index c89caa0..ffd2226 100644
--- a/mycli/myclirc
+++ b/mycli/myclirc
@@ -27,9 +27,12 @@ log_level = INFO
 # line below.
 # audit_log = ~/.mycli-audit.log
 
-# Timing of sql statments and table rendering.
+# Timing of sql statements and table rendering.
 timing = True
 
+# Beep after long-running queries are completed; 0 to disable.
+beep_after_seconds = 0
+
 # Table format. Possible values: ascii, double, github,
 # psql, plain, simple, grid, fancy_grid, pipe, orgtbl, rst, mediawiki, html,
 # latex, latex_booktabs, textile, moinmoin, jira, vertical, tsv, csv.
@@ -63,7 +66,7 @@ wider_completion_menu = False
 # \R - The current time, in 24-hour military time (0-23)
 # \r - The current time, standard 12-hour time (1-12)
 # \s - Seconds of the current time
-# \t - Product type (Percona, MySQL, MariaDB)
+# \t - Product type (Percona, MySQL, MariaDB, TiDB)
 # \A - DSN alias name (from the [alias_dsn] section)
 # \u - Username
 # \x1b[...m - insert ANSI escape sequence
diff --git a/mycli/packages/parseutils.py b/mycli/packages/parseutils.py
index d47f59a..3090530 100644
--- a/mycli/packages/parseutils.py
+++ b/mycli/packages/parseutils.py
@@ -143,7 +143,7 @@ def extract_table_identifiers(token_stream):
 
 # extract_tables is inspired from examples in the sqlparse lib.
 def extract_tables(sql):
-    """Extract the table names from an SQL statment.
+    """Extract the table names from an SQL statement.
 
     Returns a list of (schema, table, alias) tuples
 
diff --git a/mycli/packages/special/dbcommands.py b/mycli/packages/special/dbcommands.py
index 45d7069..5c29c55 100644
--- a/mycli/packages/special/dbcommands.py
+++ b/mycli/packages/special/dbcommands.py
@@ -34,6 +34,7 @@ def list_tables(cur, arg=None, arg_type=PARSED_QUERY, verbose=False):
 
     return [(None, tables, headers, status)]
 
+
 @special_command('\\l', '\\l', 'List databases.', arg_type=RAW_QUERY, case_sensitive=True)
 def list_databases(cur, **_):
     query = 'SHOW DATABASES'
@@ -45,6 +46,7 @@ def list_databases(cur, **_):
     else:
         return [(None, None, None, '')]
 
+
 @special_command('status', '\\s', 'Get status information from the server.',
                  arg_type=RAW_QUERY, aliases=('\\s', ), case_sensitive=True)
 def status(cur, **_):
@@ -146,7 +148,8 @@ def status(cur, **_):
             stats.append('Queries: {0}'.format(status['Queries']))
         stats.append('Slow queries: {0}'.format(status['Slow_queries']))
         stats.append('Opens: {0}'.format(status['Opened_tables']))
-        stats.append('Flush tables: {0}'.format(status['Flush_commands']))
+        if 'Flush_commands' in status:
+            stats.append('Flush tables: {0}'.format(status['Flush_commands']))
         stats.append('Open tables: {0}'.format(status['Open_tables']))
         if 'Queries' in status:
             queries_per_second = int(status['Queries']) / int(status['Uptime'])
diff --git a/mycli/sqlexecute.py b/mycli/sqlexecute.py
index 9461438..c019707 100644
--- a/mycli/sqlexecute.py
+++ b/mycli/sqlexecute.py
@@ -28,6 +28,7 @@ class ServerSpecies(enum.Enum):
     MySQL = 'MySQL'
     MariaDB = 'MariaDB'
     Percona = 'Percona'
+    TiDB = 'TiDB'
     Unknown = 'MySQL'
 
 
@@ -55,6 +56,7 @@ class ServerInfo:
 
         re_species = (
             (r'(?P<version>[0-9\.]+)-MariaDB', ServerSpecies.MariaDB),
+            (r'(?P<version>[0-9\.]+)[a-z0-9]*-TiDB', ServerSpecies.TiDB),
             (r'(?P<version>[0-9\.]+)[a-z0-9]*-(?P<comment>[0-9]+$)',
              ServerSpecies.Percona),
             (r'(?P<version>[0-9\.]+)[a-z0-9]*-(?P<comment>[A-Za-z0-9_]+)',
@@ -338,10 +340,16 @@ class SQLExecute(object):
     def reset_connection_id(self):
         # Remember current connection id
         _logger.debug('Get current connection id')
-        res = self.run('select connection_id()')
-        for title, cur, headers, status in res:
-            self.connection_id = cur.fetchone()[0]
-        _logger.debug('Current connection id: %s', self.connection_id)
+        try:
+            res = self.run('select connection_id()')
+            for title, cur, headers, status in res:
+                self.connection_id = cur.fetchone()[0]
+        except Exception as e:
+            # See #1054
+            self.connection_id = -1
+            _logger.error('Failed to get connection id: %s', e)
+        else:
+            _logger.debug('Current connection id: %s', self.connection_id)
 
     def change_db(self, db):
         self.conn.select_db(db)
diff --git a/release.py b/release.py
index 3f18f03..62daa80 100755
--- a/release.py
+++ b/release.py
@@ -72,7 +72,7 @@ def upload_distribution_files():
 
 
 def push_to_github():
-    run_step('git', 'push', 'origin', 'master')
+    run_step('git', 'push', 'origin', 'main')
 
 
 def push_tags_to_github():
@@ -90,7 +90,6 @@ if __name__ == '__main__':
         subprocess.check_output = lambda x: x
 
     ver = version('mycli/__init__.py')
-    print('Releasing Version:', ver)
 
     parser = OptionParser()
     parser.add_option(
@@ -107,6 +106,8 @@ if __name__ == '__main__':
     CONFIRM_STEPS = popts.confirm_steps
     DRY_RUN = popts.dry_run
 
+    print('Releasing Version:', ver)
+
     if not click.confirm('Are you sure?', default=False):
         sys.exit(1)
 
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 9c40316..3d10e9a 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,13 +1,16 @@
-pytest!=3.3.0
-pytest-cov==2.4.0
+pytest>=3.3.0
+pytest-cov>=2.4.0
 tox
-twine==1.12.1
+twine>=1.12.1
 behave>=1.2.4
-pexpect==3.3
-coverage==5.0.4
-codecov==2.0.9
+pexpect>=3.3
+coverage>=5.0.4
+codecov>=2.0.9
 autopep8==1.3.3
-colorama==0.4.1
+colorama>=0.4.1
 git+https://github.com/hayd/pep8radius.git  # --error-status option not released
 click>=7.0
-paramiko==2.7.1
+paramiko==2.11.0
+pyperclip>=1.8.1
+importlib_resources>=5.0.0
+pyaes>=1.6.1
diff --git a/setup.py b/setup.py
index f79bcd7..c9dcc44 100755
--- a/setup.py
+++ b/setup.py
@@ -18,7 +18,9 @@ description = 'CLI for MySQL Database. With auto-completion and syntax highlight
 
 install_requirements = [
     'click >= 7.0',
-    'cryptography >= 1.0.0',
+    # Temporary to suppress paramiko Blowfish warning which breaks CI.
+    # Pinning cryptography should not be needed after paramiko 2.11.0.
+    'cryptography == 36.0.2',
     # 'Pygments>=1.6,<=2.11.1',
     'Pygments>=1.6',
     'prompt_toolkit>=3.0.6,<4.0.0',
@@ -38,14 +40,14 @@ class lint(Command):
     description = 'check code against PEP 8 (and fix violations)'
 
     user_options = [
-        ('branch=', 'b', 'branch/revision to compare against (e.g. master)'),
+        ('branch=', 'b', 'branch/revision to compare against (e.g. main)'),
         ('fix', 'f', 'fix the violations in place'),
         ('error-status', 'e', 'return an error code on failed PEP check'),
     ]
 
     def initialize_options(self):
         """Set the default options."""
-        self.branch = 'master'
+        self.branch = 'main'
         self.fix = False
         self.error_status = True
 
diff --git a/test/conftest.py b/test/conftest.py
index d7d10ce..1325596 100644
--- a/test/conftest.py
+++ b/test/conftest.py
@@ -6,8 +6,8 @@ import mycli.sqlexecute
 
 @pytest.fixture(scope="function")
 def connection():
-    create_db('_test_db')
-    connection = db_connection('_test_db')
+    create_db('mycli_test_db')
+    connection = db_connection('mycli_test_db')
     yield connection
 
     connection.close()
@@ -22,7 +22,7 @@ def cursor(connection):
 @pytest.fixture
 def executor(connection):
     return mycli.sqlexecute.SQLExecute(
-        database='_test_db', user=USER,
+        database='mycli_test_db', user=USER,
         host=HOST, password=PASSWORD, port=PORT, socket=None, charset=CHARSET,
         local_infile=False, ssl=None, ssh_user=SSH_USER, ssh_host=SSH_HOST,
         ssh_port=SSH_PORT, ssh_password=None, ssh_key_filename=None
diff --git a/test/test_main.py b/test/test_main.py
index 00fdc1b..c3351ec 100644
--- a/test/test_main.py
+++ b/test/test_main.py
@@ -1,4 +1,5 @@
 import os
+import shutil
 
 import click
 from click.testing import CliRunner
@@ -24,7 +25,7 @@ os.environ['MYSQL_TEST_LOGIN_FILE'] = login_path_file
 CLI_ARGS = ['--user', USER, '--host', HOST, '--port', PORT,
             '--password', PASSWORD, '--myclirc', default_config_file,
             '--defaults-file', default_config_file,
-            '_test_db']
+            'mycli_test_db']
 
 
 @dbtest
@@ -258,13 +259,13 @@ def test_reserved_space_is_integer():
     def stub_terminal_size():
         return (5, 5)
 
-    old_func = click.get_terminal_size
+    old_func = shutil.get_terminal_size
 
-    click.get_terminal_size = stub_terminal_size
+    shutil.get_terminal_size = stub_terminal_size
     mycli = MyCli()
     assert isinstance(mycli.get_reserved_space(), int)
 
-    click.get_terminal_size = old_func
+    shutil.get_terminal_size = old_func
 
 
 def test_list_dsn():
@@ -304,19 +305,25 @@ def test_dsn(monkeypatch):
     # Setup classes to mock mycli.main.MyCli
     class Formatter:
         format_name = None
+
     class Logger:
         def debug(self, *args, **args_dict):
             pass
+
         def warning(self, *args, **args_dict):
             pass
+
     class MockMyCli:
         config = {'alias_dsn': {}}
+
         def __init__(self, **args):
             self.logger = Logger()
             self.destructive_warning = False
             self.formatter = Formatter()
+
         def connect(self, **args):
             MockMyCli.connect_args = args
+
         def run_query(self, query, new_line=True):
             pass
 
diff --git a/test/test_sqlexecute.py b/test/test_sqlexecute.py
index 0f38a97..38ca5ef 100644
--- a/test/test_sqlexecute.py
+++ b/test/test_sqlexecute.py
@@ -71,7 +71,7 @@ def test_table_and_columns_query(executor):
 @dbtest
 def test_database_list(executor):
     databases = executor.databases()
-    assert '_test_db' in databases
+    assert 'mycli_test_db' in databases
 
 
 @dbtest
@@ -276,6 +276,7 @@ def test_multiple_results(executor):
 @pytest.mark.parametrize(
     'version_string, species, parsed_version_string, version',
     (
+        ('5.7.25-TiDB-v6.1.0','TiDB', '5.7.25', 50725),
         ('5.7.32-35', 'Percona', '5.7.32', 50732),
         ('5.7.32-0ubuntu0.18.04.1', 'MySQL', '5.7.32', 50732),
         ('10.5.8-MariaDB-1:10.5.8+maria~focal', 'MariaDB', '10.5.8', 100508),
diff --git a/test/utils.py b/test/utils.py
index 66b4194..ab12248 100644
--- a/test/utils.py
+++ b/test/utils.py
@@ -41,8 +41,8 @@ dbtest = pytest.mark.skipif(
 def create_db(dbname):
     with db_connection().cursor() as cur:
         try:
-            cur.execute('''DROP DATABASE IF EXISTS _test_db''')
-            cur.execute('''CREATE DATABASE _test_db''')
+            cur.execute('''DROP DATABASE IF EXISTS mycli_test_db''')
+            cur.execute('''CREATE DATABASE mycli_test_db''')
         except:
             pass