hohn/codeql
1
0
Fork 0
mirror of https://github.com/github/codeql.git synced 2025-12-16 16:53:25 +01:00
Code Issues Packages Projects Releases Wiki Activity
84,884 Commits 1,671 Branches 157 Tags
main
Commit Graph

80 Commits

Author SHA1 Message Date
Paolo Tranquilli
261c129555 MaD generator: add single file mode 2025-06-19 12:57:05 +02:00
Paolo Tranquilli
2818e6ee17 MaD generator: switch to snake_case fields 2025-06-19 11:59:28 +02:00
Paolo Tranquilli
5df292c286 MaD generator: apply black formatting to all sources 2025-06-13 08:47:07 +02:00
Paolo Tranquilli
1a36374718 MaD generator: really fix ordering problem 2025-06-13 08:45:55 +02:00
Paolo Tranquilli
f7266c973b Merge branch 'main' into redsun82/mad-generator 2025-06-13 08:42:06 +02:00
Paolo Tranquilli
ae3bbb0a9b MaD generator: run black formatter` 2025-06-12 16:52:29 +02:00
Paolo Tranquilli
39a3623b18 MaD generator: change default thread and ram 
The standalone MaD generator now uses `0` for threads and throttles the
RAM to use 2GB per thread by default.

Also, replaced the hand-written argument parsing with `argparse`.
 2025-06-12 16:52:14 +02:00
Paolo Tranquilli
7db31b06d1 MaD generator: tweak the scripts 
* fix a bug where the order of model generation was determined by the
  order in the `download.json` file of the experiment rather than the
  order in the config file
* allow configuring `--ram` and `--threads` in the MaD generator scripts
* use no `--ram` and `--threads=0` by default in the bulk generator
  (single generator defaults are left unchanged)
* allow to pass `--dca` multiple times, taking DBs from experiments
  listed last. This allows to run a subset of the sources in a "fixup"
  experiment and use it to "patch" a previous run without rerunning
  everything.
 2025-06-12 16:23:07 +02:00
Paolo Tranquilli
c6c52edbee MaD generator: tweak the scripts 
* fix a bug where the order of model generation was determined by the
  order in the `download.json` file of the experiment rather than the
  order in the config file
* allow configuring `--ram` and `--threads` in the MaD generator scripts
* use no `--ram` and `--threads=0` by default in the bulk generator
  (single generator defaults are left unchanged)
* allow to pass `--dca` multiple times, taking DBs from experiments
  listed last. This allows to run a subset of the sources in a "fixup"
  experiment and use it to "patch" a previous run without rerunning
  everything.
 2025-06-11 16:42:00 +02:00
Paolo Tranquilli
051704e59d Merge branch 'main' into redsun82/rust-mad 2025-06-11 16:39:10 +02:00
Paolo Tranquilli
ca99add7d6 MaD generator: address review 2025-06-10 12:14:39 +02:00
Paolo Tranquilli
ecc35e5a07 MaD generator: run formatter 2025-06-10 12:11:00 +02:00
Paolo Tranquilli
d6d13b9164 MaD generator: use decompress terminology instead of extract 2025-06-06 17:36:05 +02:00
Paolo Tranquilli
e1eb1f6345 Rust: address review 
Also fix some minor things in `bulk_generate_mad.py`.
 2025-06-05 16:03:40 +02:00
Paolo Tranquilli
530b990dd5 MaD generator: some final minor tweaks 2025-06-05 10:08:01 +02:00
Paolo Tranquilli
ee7eb86a1d MaD: make bulk generator cleanup downloaded DBs 2025-06-05 09:31:24 +02:00
Paolo Tranquilli
4f47ee2e72 MaD: make bulk generator DCA strategy download DBs in parallel 2025-06-05 09:30:02 +02:00
Paolo Tranquilli
31954fa794 MaD generator: make bulk generator executable 2025-06-05 08:49:15 +02:00
Paolo Tranquilli
d5c16d6092 MaD generator: reformat 2025-06-05 08:48:01 +02:00
Paolo Tranquilli
900a3b0992 MaD generator: only pick up last database on comparison DCAs 2025-06-05 08:47:45 +02:00
Paolo Tranquilli
31d1604337 Bulk model generator: switch from json to yml configuration files 2025-06-05 08:37:43 +02:00
Paolo Tranquilli
5f474a7185 Rust: Use QL computed canonical paths in MaD Field tokens 
Also regenerate all auto-generated models with latest extractor.
 2025-06-04 09:43:45 +02:00
Mathias Vorreiter Pedersen
7cb9024cc6 Bulk generator: Flip default values for summaries, sources, and sinks. 2025-05-30 13:33:24 +01:00
Mathias Vorreiter Pedersen
0f30644afd Bulk generator: Snake case things. 2025-05-30 13:26:53 +01:00
Mathias Vorreiter Pedersen
3444c986ec Bulk generator: Fix field name. 2025-05-30 13:25:12 +01:00
Mathias Vorreiter Pedersen
bdf411afbc Bulk generator: Make 'database_results' a map to simplify away the explicit sorting. 2025-05-30 13:09:55 +01:00
Mathias Vorreiter Pedersen
cdd869a970 Bulk generator: Autoformat. 2025-05-30 12:49:12 +01:00
Mathias Vorreiter Pedersen
3ddca32705 Update misc/scripts/models-as-data/bulk_generate_mad.py 
Co-authored-by: Simon Friis Vindum <paldepind@github.com>
 2025-05-30 12:48:50 +01:00
Mathias Vorreiter Pedersen
7c2612a6a1 Bulk generator: Specify a path to the PAT instead of the PAT itself. 2025-05-30 12:47:07 +01:00
Mathias Vorreiter Pedersen
1228080914 Bulk generator: Specify 'language' in the config file. 2025-05-30 12:40:21 +01:00
Mathias Vorreiter Pedersen
fc165db8ac Bulk generator: Specify 'with-summaries', 'with-sources', and 'with-sinks' in the config file. 2025-05-30 12:40:20 +01:00
Mathias Vorreiter Pedersen
7121f5c57e Bulk generator: Use the 'Project' type throughout the file. 2025-05-30 12:08:42 +01:00
Mathias Vorreiter Pedersen
7c89d6d6dd Bulk generator: Rename 'get_destination_for_project' to 'get_mad_destination_for_project'. 2025-05-30 11:49:48 +01:00
Mathias Vorreiter Pedersen
5d79a8de89 Update misc/scripts/models-as-data/bulk_generate_mad.py 
Co-authored-by: Simon Friis Vindum <paldepind@github.com>
 2025-05-30 11:48:30 +01:00
Mathias Vorreiter Pedersen
b640474a61 Bulk generator: Remove 'Phase' part of log message. 2025-05-30 11:43:30 +01:00
Mathias Vorreiter Pedersen
566bf431d7 Bulk generator: Rename 'github' to 'get_json_from_github'. 2025-05-30 11:42:15 +01:00
Mathias Vorreiter Pedersen
7ecf8c8ea2 Bulk generator: Format file and add a note at the top of the file specifying the formatting requirements. 2025-05-30 11:40:55 +01:00
Mathias Vorreiter Pedersen
cb938701a1 Bulk generator: Rename file since it is no longer Rust specific. 2025-05-29 18:14:24 +01:00
Mathias Vorreiter Pedersen
5051790e24 Bulk generator: Add DCA support. 2025-05-29 18:14:22 +01:00
Mathias Vorreiter Pedersen
e721fc07aa Bulk generator: Prepare for adding DCA support. This commits just generalizes the existing functionality to be independent of Rust and instead depend on the configuration file and the command-line arguments. 2025-05-29 18:14:21 +01:00
Mathias Vorreiter Pedersen
6ff2bebbc2 Bulk generator: Add command-line arguments. 2025-05-29 18:14:19 +01:00
Mathias Vorreiter Pedersen
b87ba31c43 Bulk generator: Get rid of the hardcoded project list and move it into a configuration file. 2025-05-29 18:14:18 +01:00
Mathias Vorreiter Pedersen
40d937a2eb Bulk generator: Some imports we will need. 2025-05-29 18:14:16 +01:00
Simon Friis Vindum
d66c12b7a9 Rust: Add MaD bulk generation script 2025-05-16 13:38:24 +02:00
Simon Friis Vindum
14bdc1ab22 Shared: Minor tweaks to model generator script 
Co-authored-by: Michael Nebel <michaelnebel@github.com>
 2025-05-13 09:31:50 +02:00
Simon Friis Vindum
c6d95ceeb0 Shared: Remove the language-specific model generator scripts 2025-05-02 13:21:10 +02:00
Michael Nebel
2357a69d55 Shared: Remove the --with-mixed-neutrals logic. 2025-04-25 15:49:44 +02:00
Michael Nebel
ae70c76a07 Shared: Use the CaptureSummaryModels instead of CaptureMixedSummaryModels. 2025-04-25 15:49:42 +02:00
Michael Nebel
fa5162fb13 Shared: Remove the backwards compatbility flag. 2025-04-25 15:49:41 +02:00
Michael Nebel
2a8fe53b04 Shared: Remove --with-mixed-summaries logic. 2025-04-25 15:49:38 +02:00