Merge pull request #8172 from github/redsun82/pre-commit

add pre-commit configuration

.pre-commit-config.yaml

@@ -0,0 +1,29 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+exclude: /test/.*$(?<!\.ql)(?<!\.qll)(?<!\.qlref)
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.2.0
+    hooks:
+    -   id: trailing-whitespace
+    -   id: end-of-file-fixer
+
+-   repo: local
+    hooks:
+    -   id: codeql-format
+        name: Fix QL file formatting
+        files: \.qll?$
+        language: system
+        entry: codeql query format --in-place
+
+    -   id: sync-files
+        name: Fix files required to be identical
+        language: system
+        entry: python3 config/sync-files.py --latest
+        pass_filenames: false
+
+    -   id: qhelp
+        name: Check query help generation
+        files: \.qhelp$
+        language: system
+        entry: python3 misc/scripts/check-qhelp.py

CONTRIBUTING.md

@@ -42,7 +42,11 @@ If you have an idea for a query that you would like to share with other CodeQL u
 
     - The queries and libraries must be autoformatted, for example using the "Format Document" command in [CodeQL for Visual Studio Code](https://help.semmle.com/codeql/codeql-for-vscode/procedures/about-codeql-for-vscode.html).
 
-    If you prefer, you can use this [pre-commit hook](misc/scripts/pre-commit) that automatically checks whether your files are correctly formatted. See the [pre-commit hook installation guide](docs/pre-commit-hook-setup.md) for instructions on how to install the hook.
+    If you prefer, you can either:
+    1. install the [pre-commit framework](https://pre-commit.com/) and install the configured hooks on this repo via `pre-commit install`, or
+    2. use this [pre-commit hook](misc/scripts/pre-commit) that automatically checks whether your files are correctly formatted.
+
+    See the [pre-commit hook installation guide](docs/pre-commit-hook-setup.md) for instructions on the two approaches.
 
 4. **Compilation**
 
@@ -63,6 +67,6 @@ After the experimental query is merged, we welcome pull requests to improve it.
 
 ## Using your personal data
 
-If you contribute to this project, we will record your name and email address (as provided by you with your contributions) as part of the code repositories, which are public. We might also use this information to contact you in relation to your contributions, as well as in the normal course of software development. We also store records of CLA agreements signed in the past, but no longer require contributors to sign a CLA. Under GDPR legislation, we do this on the basis of our legitimate interest in creating the CodeQL product. 
+If you contribute to this project, we will record your name and email address (as provided by you with your contributions) as part of the code repositories, which are public. We might also use this information to contact you in relation to your contributions, as well as in the normal course of software development. We also store records of CLA agreements signed in the past, but no longer require contributors to sign a CLA. Under GDPR legislation, we do this on the basis of our legitimate interest in creating the CodeQL product.
 
 Please do get in touch (privacy@github.com) if you have any questions about this or our data protection policies.

docs/pre-commit-hook-setup.md

@@ -1,6 +1,44 @@
 # CodeQL pre-commit-hook setup
 
-As stated in [CONTRIBUTING](../CONTRIBUTING.md) all CodeQL files must be formatted according to our [CodeQL style guide](ql-style-guide.md). You can use our pre-commit hook to avoid committing incorrectly formatted code. To use it, simply copy the [pre-commit](../misc/scripts/pre-commit) script to `.git/hooks/pre-commit` and make sure that:
+
+As stated in [CONTRIBUTING](../CONTRIBUTING.md) all CodeQL files must be formatted according to our [CodeQL style guide](ql-style-guide.md). You can use a pre-commit hook to avoid committing incorrectly formatted code, as well as prevent some other easily checkable errors.
+
+## Using the `pre-commit` framework
+
+Preferably, you can use the [pre-commit framework](https://pre-commit.com/). There are some pre-commit hooks already configured on [`.pre-commit-config.yaml`](../.pre-commit-config.yaml). In order to install them you need to follow pre-commit's [installation instructions](https://pre-commit.com/#installation) and then run `pre-commit install`. Typically (assuming you have [`pip`](https://pip.pypa.io/en/stable/installation/) installed):
+```
+python3 -m pip install pre-commit
+pre-commit install
+```
+
+Also, make sure that the CodeQL CLI has been added to your `PATH`.
+
+By default, pre-commit will check and fix:
+* trailing whitespaces;
+* absence of or duplicate newlines at end of files;
+* QL formatting;
+* files out of sync (see [`config/sync-files.py`](../config/sync-files.py)).
+
+It will additionally check:
+* `.qhelp` files for query help generation.
+
+It will run the checks only on files changed by the commit (except for the file sync check) and it will skip all files under `test` directories unless they are `.ql`, `.qll` or `.qlref` files.
+
+If you want to change any behaviour (for example, you want to skip the out-of-sync file check, or you want to avoid auto-fixing formatting or file syncing), you can copy the configuration file to a separate location, modify it and use that. For example
+```
+cp .pre-commit-config.yaml ~/my-codeql-pre-commit-config.yaml
+pre-commit install --config ~/my-codeql-pre-commit-config.yaml
+# edit ~/my-codeql-pre-commit-config.yaml to your liking
+```
+
+You can for example:
+* change `--in-place` to `--check-only` in the `codeql-format` hook to have it report formatting problems instead of auto-fixing them;
+* remove `--latest` in the `sync-files` hook to do the same;
+* remove any hook altogether.
+
+## Manual approach
+
+You can have the formatting check in place by copying the [pre-commit](../misc/scripts/pre-commit) script to `.git/hooks/pre-commit` and make sure that:
 
 - The script is executable. On Linux and macOS this can be done using `chmod +x`.
 - The CodeQL CLI has been added to your `PATH`.

misc/scripts/check-qhelp.py

@@ -0,0 +1,70 @@
+#!/bin/env python3
+
+"""cross platform wrapper around codeql generate query-help to check .qhelp files
+
+This takes care of:
+* providing a temporary directory to --output
+* finding usages of .inc.qhelp arguments
+"""
+
+import pathlib
+import tempfile
+import sys
+import subprocess
+import xml.sax
+
+
+include_cache = {}
+
+class IncludeHandler(xml.sax.ContentHandler):
+    def __init__(self, xml):
+        self.__xml = xml
+
+    def startElement(self, name, attrs):
+        if name == "include":
+            src = (self.__xml.parent / attrs["src"]).resolve()
+            include_cache.setdefault(src, set()).add(self.__xml)
+
+class IgnoreErrorsHandler(xml.sax.ErrorHandler):
+    def error(self, exc):
+        pass
+
+    def fatalError(self, exc):
+        pass
+
+    def warning(self, exc):
+        pass
+
+def init_include_cache():
+    if not include_cache:
+        for qhelp in pathlib.Path().rglob("*.qhelp"):
+            xml.sax.parse(qhelp, IncludeHandler(qhelp), IgnoreErrorsHandler())
+
+
+def find_inc_qhelp_usages(arg):
+    init_include_cache()
+    return include_cache.get(arg.resolve(), ())
+
+def transform_inputs(args):
+    for arg in args:
+        arg = pathlib.Path(arg)
+        if arg.suffixes == ['.inc', '.qhelp']:
+            for qhelp in find_inc_qhelp_usages(arg):
+                yield str(qhelp)
+        else:
+            yield str(arg)
+
+affected_qhelp_files = list(transform_inputs(sys.argv[1:]))
+if not affected_qhelp_files:
+    # can happen with changes on an unused .inc.qhelp file
+    print("nothing to do!")
+    sys.exit(0)
+
+cmd = ["codeql", "generate", "query-help", "--format=markdown"]
+
+with tempfile.TemporaryDirectory() as tmp:
+    cmd += [f"--output={tmp}", "--"]
+    cmd += affected_qhelp_files
+    res = subprocess.run(cmd)
+
+sys.exit(res.returncode)