Initial Vexta Core v0.1.0 release
This commit is contained in:
commit
25c454329d
2481 changed files with 689289 additions and 0 deletions
180
contrib/devtools/README.md
Normal file
180
contrib/devtools/README.md
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
Contents
|
||||
========
|
||||
This directory contains tools for developers working on this repository.
|
||||
|
||||
clang-format-diff.py
|
||||
===================
|
||||
|
||||
A script to format unified git diffs according to [.clang-format](../../src/.clang-format).
|
||||
|
||||
Requires `clang-format`, installed e.g. via `brew install clang-format` on macOS,
|
||||
or `sudo apt install clang-format` on Debian/Ubuntu.
|
||||
|
||||
For instance, to format the last commit with 0 lines of context,
|
||||
the script should be called from the git root folder as follows.
|
||||
|
||||
```
|
||||
git diff -U0 HEAD~1.. | ./contrib/devtools/clang-format-diff.py -p1 -i -v
|
||||
```
|
||||
|
||||
copyright\_header.py
|
||||
====================
|
||||
|
||||
Provides utilities for managing copyright headers of `The DigiByte Core
|
||||
developers` in repository source files. It has three subcommands:
|
||||
|
||||
```
|
||||
$ ./copyright_header.py report <base_directory> [verbose]
|
||||
$ ./copyright_header.py update <base_directory>
|
||||
$ ./copyright_header.py insert <file>
|
||||
```
|
||||
Running these subcommands without arguments displays a usage string.
|
||||
|
||||
copyright\_header.py report \<base\_directory\> [verbose]
|
||||
---------------------------------------------------------
|
||||
|
||||
Produces a report of all copyright header notices found inside the source files
|
||||
of a repository. Useful to quickly visualize the state of the headers.
|
||||
Specifying `verbose` will list the full filenames of files of each category.
|
||||
|
||||
copyright\_header.py update \<base\_directory\> [verbose]
|
||||
---------------------------------------------------------
|
||||
Updates all the copyright headers of `The DigiByte Core developers` which were
|
||||
changed in a year more recent than is listed. For example:
|
||||
```
|
||||
// Copyright (c) <firstYear>-<lastYear> The DigiByte Core developers
|
||||
```
|
||||
will be updated to:
|
||||
```
|
||||
// Copyright (c) <firstYear>-<lastModifiedYear> The DigiByte Core developers
|
||||
```
|
||||
where `<lastModifiedYear>` is obtained from the `git log` history.
|
||||
|
||||
This subcommand also handles copyright headers that have only a single year. In
|
||||
those cases:
|
||||
```
|
||||
// Copyright (c) <year> The DigiByte Core developers
|
||||
```
|
||||
will be updated to:
|
||||
```
|
||||
// Copyright (c) <year>-<lastModifiedYear> The DigiByte Core developers
|
||||
```
|
||||
where the update is appropriate.
|
||||
|
||||
copyright\_header.py insert \<file\>
|
||||
------------------------------------
|
||||
Inserts a copyright header for `The DigiByte Core developers` at the top of the
|
||||
file in either Python or C++ style as determined by the file extension. If the
|
||||
file is a Python file and it has `#!` starting the first line, the header is
|
||||
inserted in the line below it.
|
||||
|
||||
The copyright dates will be set to be `<year_introduced>-<current_year>` where
|
||||
`<year_introduced>` is according to the `git log` history. If
|
||||
`<year_introduced>` is equal to `<current_year>`, it will be set as a single
|
||||
year rather than two hyphenated years.
|
||||
|
||||
If the file already has a copyright for `The DigiByte Core developers`, the
|
||||
script will exit.
|
||||
|
||||
gen-manpages.sh
|
||||
===============
|
||||
|
||||
A small script to automatically create manpages in ../../doc/man by running the release binaries with the -help option.
|
||||
This requires help2man which can be found at: https://www.gnu.org/software/help2man/
|
||||
|
||||
With in-tree builds this tool can be run from any directory within the
|
||||
repostitory. To use this tool with out-of-tree builds set `BUILDDIR`. For
|
||||
example:
|
||||
|
||||
```bash
|
||||
BUILDDIR=$PWD/build contrib/devtools/gen-manpages.sh
|
||||
```
|
||||
|
||||
github-merge.py
|
||||
===============
|
||||
|
||||
A small script to automate merging pull-requests securely and sign them with GPG.
|
||||
|
||||
For example:
|
||||
|
||||
./github-merge.py 3077
|
||||
|
||||
(in any git repository) will help you merge pull request #3077 for the
|
||||
digibyte/digibyte repository.
|
||||
|
||||
What it does:
|
||||
* Fetch master and the pull request.
|
||||
* Locally construct a merge commit.
|
||||
* Show the diff that merge results in.
|
||||
* Ask you to verify the resulting source tree (so you can do a make
|
||||
check or whatever).
|
||||
* Ask you whether to GPG sign the merge commit.
|
||||
* Ask you whether to push the result upstream.
|
||||
|
||||
This means that there are no potential race conditions (where a
|
||||
pullreq gets updated while you're reviewing it, but before you click
|
||||
merge), and when using GPG signatures, that even a compromised GitHub
|
||||
couldn't mess with the sources.
|
||||
|
||||
Setup
|
||||
---------
|
||||
Configuring the github-merge tool for the digibyte repository is done in the following way:
|
||||
|
||||
git config githubmerge.repository digibyte/digibyte
|
||||
git config githubmerge.testcmd "make -j4 check" (adapt to whatever you want to use for testing)
|
||||
git config --global user.signingkey mykeyid (if you want to GPG sign)
|
||||
|
||||
Create and verify timestamps of merge commits
|
||||
---------------------------------------------
|
||||
To create or verify timestamps on the merge commits, install the OpenTimestamps
|
||||
client via `pip3 install opentimestamps-client`. Then, dowload the gpg wrapper
|
||||
`ots-git-gpg-wrapper.sh` and set it as git's `gpg.program`. See
|
||||
[the ots git integration documentation](https://github.com/opentimestamps/opentimestamps-client/blob/master/doc/git-integration.md#usage)
|
||||
for further details.
|
||||
|
||||
optimize-pngs.py
|
||||
================
|
||||
|
||||
A script to optimize png files in the digibyte
|
||||
repository (requires pngcrush).
|
||||
|
||||
=======
|
||||
security-check.py and test-security-check.py
|
||||
============================================
|
||||
|
||||
Perform basic security checks on a series of executables.
|
||||
|
||||
symbol-check.py
|
||||
===============
|
||||
|
||||
A script to check that the executables produced by gitian only contain
|
||||
certain symbols and are only linked against allowed libraries.
|
||||
|
||||
For Linux this means checking for allowed gcc, glibc and libstdc++ version symbols.
|
||||
This makes sure they are still compatible with the minimum supported distribution versions.
|
||||
|
||||
For macOS and Windows we check that the executables are only linked against libraries we allow.
|
||||
|
||||
Example usage after a gitian build:
|
||||
|
||||
find ../gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
|
||||
|
||||
If no errors occur the return value will be 0 and the output will be empty.
|
||||
|
||||
If there are any errors the return value will be 1 and output like this will be printed:
|
||||
|
||||
.../64/test_digibyte: symbol memcpy from unsupported version GLIBC_2.14
|
||||
.../64/test_digibyte: symbol __fdelt_chk from unsupported version GLIBC_2.15
|
||||
.../64/test_digibyte: symbol std::out_of_range::~out_of_range() from unsupported version GLIBCXX_3.4.15
|
||||
.../64/test_digibyte: symbol _ZNSt8__detail15_List_nod from unsupported version GLIBCXX_3.4.15
|
||||
|
||||
circular-dependencies.py
|
||||
========================
|
||||
|
||||
Run this script from the root of the source tree (`src/`) to find circular dependencies in the source code.
|
||||
This looks only at which files include other files, treating the `.cpp` and `.h` file as one unit.
|
||||
|
||||
Example usage:
|
||||
|
||||
cd .../src
|
||||
../contrib/devtools/circular-dependencies.py {*,*/*,*/*/*}.{h,cpp}
|
||||
46
contrib/devtools/check-doc.py
Executable file
46
contrib/devtools/check-doc.py
Executable file
|
|
@ -0,0 +1,46 @@
|
|||
#!/usr/bin/env python
|
||||
# Copyright (c) 2015-2017 The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
'''
|
||||
This checks if all command line args are documented.
|
||||
Return value is 0 to indicate no error.
|
||||
|
||||
Author: @MarcoFalke
|
||||
'''
|
||||
|
||||
from subprocess import check_output
|
||||
import re
|
||||
import sys
|
||||
|
||||
FOLDER_GREP = 'src'
|
||||
FOLDER_TEST = 'src/test/'
|
||||
CMD_ROOT_DIR = '`git rev-parse --show-toplevel`/%s' % FOLDER_GREP
|
||||
CMD_GREP_ARGS = r"egrep -r -I '(map(Multi)?Args(\.count\(|\[)|Get(Bool)?Arg\()\"\-[^\"]+?\"' %s | grep -v '%s'" % (CMD_ROOT_DIR, FOLDER_TEST)
|
||||
CMD_GREP_DOCS = r"egrep -r -I 'HelpMessageOpt\(\"\-[^\"=]+?(=|\")' %s" % (CMD_ROOT_DIR)
|
||||
REGEX_ARG = re.compile(r'(?:map(?:Multi)?Args(?:\.count\(|\[)|Get(?:Bool)?Arg\()\"(\-[^\"]+?)\"')
|
||||
REGEX_DOC = re.compile(r'HelpMessageOpt\(\"(\-[^\"=]+?)(?:=|\")')
|
||||
# list unsupported, deprecated and duplicate args as they need no documentation
|
||||
SET_DOC_OPTIONAL = set(['-rpcssl', '-benchmark', '-h', '-help', '-socks', '-tor', '-debugnet', '-whitelistalwaysrelay', '-prematurewitness', '-walletprematurewitness', '-promiscuousmempoolflags', '-blockminsize', '-dbcrashratio', '-forcecompactdb', '-usehd'])
|
||||
|
||||
def main():
|
||||
used = check_output(CMD_GREP_ARGS, shell=True)
|
||||
docd = check_output(CMD_GREP_DOCS, shell=True)
|
||||
|
||||
args_used = set(re.findall(REGEX_ARG,used))
|
||||
args_docd = set(re.findall(REGEX_DOC,docd)).union(SET_DOC_OPTIONAL)
|
||||
args_need_doc = args_used.difference(args_docd)
|
||||
args_unknown = args_docd.difference(args_used)
|
||||
|
||||
print "Args used : %s" % len(args_used)
|
||||
print "Args documented : %s" % len(args_docd)
|
||||
print "Args undocumented: %s" % len(args_need_doc)
|
||||
print args_need_doc
|
||||
print "Args unknown : %s" % len(args_unknown)
|
||||
print args_unknown
|
||||
|
||||
sys.exit(len(args_need_doc))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
92
contrib/devtools/circular-dependencies.py
Executable file
92
contrib/devtools/circular-dependencies.py
Executable file
|
|
@ -0,0 +1,92 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2018-2020 The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
import sys
|
||||
import re
|
||||
from typing import Dict, List, Set
|
||||
|
||||
MAPPING = {
|
||||
'core_read.cpp': 'core_io.cpp',
|
||||
'core_write.cpp': 'core_io.cpp',
|
||||
}
|
||||
|
||||
# Directories with header-based modules, where the assumption that .cpp files
|
||||
# define functions and variables declared in corresponding .h files is
|
||||
# incorrect.
|
||||
HEADER_MODULE_PATHS = [
|
||||
'interfaces/'
|
||||
]
|
||||
|
||||
def module_name(path):
|
||||
if path in MAPPING:
|
||||
path = MAPPING[path]
|
||||
if any(path.startswith(dirpath) for dirpath in HEADER_MODULE_PATHS):
|
||||
return path
|
||||
if path.endswith(".h"):
|
||||
return path[:-2]
|
||||
if path.endswith(".c"):
|
||||
return path[:-2]
|
||||
if path.endswith(".cpp"):
|
||||
return path[:-4]
|
||||
return None
|
||||
|
||||
files = dict()
|
||||
deps: Dict[str, Set[str]] = dict()
|
||||
|
||||
RE = re.compile("^#include <(.*)>")
|
||||
|
||||
# Iterate over files, and create list of modules
|
||||
for arg in sys.argv[1:]:
|
||||
module = module_name(arg)
|
||||
if module is None:
|
||||
print("Ignoring file %s (does not constitute module)\n" % arg)
|
||||
else:
|
||||
files[arg] = module
|
||||
deps[module] = set()
|
||||
|
||||
# Iterate again, and build list of direct dependencies for each module
|
||||
# TODO: implement support for multiple include directories
|
||||
for arg in sorted(files.keys()):
|
||||
module = files[arg]
|
||||
with open(arg, 'r', encoding="utf8") as f:
|
||||
for line in f:
|
||||
match = RE.match(line)
|
||||
if match:
|
||||
include = match.group(1)
|
||||
included_module = module_name(include)
|
||||
if included_module is not None and included_module in deps and included_module != module:
|
||||
deps[module].add(included_module)
|
||||
|
||||
# Loop to find the shortest (remaining) circular dependency
|
||||
have_cycle: bool = False
|
||||
while True:
|
||||
shortest_cycle = None
|
||||
for module in sorted(deps.keys()):
|
||||
# Build the transitive closure of dependencies of module
|
||||
closure: Dict[str, List[str]] = dict()
|
||||
for dep in deps[module]:
|
||||
closure[dep] = []
|
||||
while True:
|
||||
old_size = len(closure)
|
||||
old_closure_keys = sorted(closure.keys())
|
||||
for src in old_closure_keys:
|
||||
for dep in deps[src]:
|
||||
if dep not in closure:
|
||||
closure[dep] = closure[src] + [src]
|
||||
if len(closure) == old_size:
|
||||
break
|
||||
# If module is in its own transitive closure, it's a circular dependency; check if it is the shortest
|
||||
if module in closure and (shortest_cycle is None or len(closure[module]) + 1 < len(shortest_cycle)):
|
||||
shortest_cycle = [module] + closure[module]
|
||||
if shortest_cycle is None:
|
||||
break
|
||||
# We have the shortest circular dependency; report it
|
||||
module = shortest_cycle[0]
|
||||
print("Circular dependency: %s" % (" -> ".join(shortest_cycle + [module])))
|
||||
# And then break the dependency to avoid repeating in other cycles
|
||||
deps[shortest_cycle[-1]] = deps[shortest_cycle[-1]] - set([module])
|
||||
have_cycle = True
|
||||
|
||||
sys.exit(1 if have_cycle else 0)
|
||||
166
contrib/devtools/clang-format-diff.py
Executable file
166
contrib/devtools/clang-format-diff.py
Executable file
|
|
@ -0,0 +1,166 @@
|
|||
#!/usr/bin/env python3
|
||||
#
|
||||
#===- clang-format-diff.py - ClangFormat Diff Reformatter ----*- python -*--===#
|
||||
#
|
||||
# The LLVM Compiler Infrastructure
|
||||
#
|
||||
# This file is distributed under the University of Illinois Open Source
|
||||
# License.
|
||||
#
|
||||
# ============================================================
|
||||
#
|
||||
# University of Illinois/NCSA
|
||||
# Open Source License
|
||||
#
|
||||
# Copyright (c) 2007-2015 University of Illinois at Urbana-Champaign.
|
||||
# All rights reserved.
|
||||
#
|
||||
# Developed by:
|
||||
#
|
||||
# LLVM Team
|
||||
#
|
||||
# University of Illinois at Urbana-Champaign
|
||||
#
|
||||
# http://llvm.org
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
# this software and associated documentation files (the "Software"), to deal with
|
||||
# the Software without restriction, including without limitation the rights to
|
||||
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
# of the Software, and to permit persons to whom the Software is furnished to do
|
||||
# so, subject to the following conditions:
|
||||
#
|
||||
# * Redistributions of source code must retain the above copyright notice,
|
||||
# this list of conditions and the following disclaimers.
|
||||
#
|
||||
# * Redistributions in binary form must reproduce the above copyright notice,
|
||||
# this list of conditions and the following disclaimers in the
|
||||
# documentation and/or other materials provided with the distribution.
|
||||
#
|
||||
# * Neither the names of the LLVM Team, University of Illinois at
|
||||
# Urbana-Champaign, nor the names of its contributors may be used to
|
||||
# endorse or promote products derived from this Software without specific
|
||||
# prior written permission.
|
||||
#
|
||||
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
# CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE
|
||||
# SOFTWARE.
|
||||
#
|
||||
# ============================================================
|
||||
#
|
||||
#===------------------------------------------------------------------------===#
|
||||
|
||||
r"""
|
||||
ClangFormat Diff Reformatter
|
||||
============================
|
||||
|
||||
This script reads input from a unified diff and reformats all the changed
|
||||
lines. This is useful to reformat all the lines touched by a specific patch.
|
||||
Example usage for git/svn users:
|
||||
|
||||
git diff -U0 HEAD^ | clang-format-diff.py -p1 -i
|
||||
svn diff --diff-cmd=diff -x-U0 | clang-format-diff.py -i
|
||||
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import difflib
|
||||
import io
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
# Change this to the full path if clang-format is not on the path.
|
||||
binary = 'clang-format'
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=
|
||||
'Reformat changed lines in diff. Without -i '
|
||||
'option just output the diff that would be '
|
||||
'introduced.')
|
||||
parser.add_argument('-i', action='store_true', default=False,
|
||||
help='apply edits to files instead of displaying a diff')
|
||||
parser.add_argument('-p', metavar='NUM', default=0,
|
||||
help='strip the smallest prefix containing P slashes')
|
||||
parser.add_argument('-regex', metavar='PATTERN', default=None,
|
||||
help='custom pattern selecting file paths to reformat '
|
||||
'(case sensitive, overrides -iregex)')
|
||||
parser.add_argument('-iregex', metavar='PATTERN', default=
|
||||
r'.*\.(cpp|cc|c\+\+|cxx|c|cl|h|hpp|m|mm|inc|js|ts|proto'
|
||||
r'|protodevel|java)',
|
||||
help='custom pattern selecting file paths to reformat '
|
||||
'(case insensitive, overridden by -regex)')
|
||||
parser.add_argument('-sort-includes', action='store_true', default=False,
|
||||
help='let clang-format sort include blocks')
|
||||
parser.add_argument('-v', '--verbose', action='store_true',
|
||||
help='be more verbose, ineffective without -i')
|
||||
args = parser.parse_args()
|
||||
|
||||
# Extract changed lines for each file.
|
||||
filename = None
|
||||
lines_by_file = {}
|
||||
for line in sys.stdin:
|
||||
match = re.search(r'^\+\+\+\ (.*?/){%s}(\S*)' % args.p, line)
|
||||
if match:
|
||||
filename = match.group(2)
|
||||
if filename is None:
|
||||
continue
|
||||
|
||||
if args.regex is not None:
|
||||
if not re.match('^%s$' % args.regex, filename):
|
||||
continue
|
||||
else:
|
||||
if not re.match('^%s$' % args.iregex, filename, re.IGNORECASE):
|
||||
continue
|
||||
|
||||
match = re.search(r'^@@.*\+(\d+)(,(\d+))?', line)
|
||||
if match:
|
||||
start_line = int(match.group(1))
|
||||
line_count = 1
|
||||
if match.group(3):
|
||||
line_count = int(match.group(3))
|
||||
if line_count == 0:
|
||||
continue
|
||||
end_line = start_line + line_count - 1
|
||||
lines_by_file.setdefault(filename, []).extend(
|
||||
['-lines', str(start_line) + ':' + str(end_line)])
|
||||
|
||||
# Reformat files containing changes in place.
|
||||
for filename, lines in lines_by_file.items():
|
||||
if args.i and args.verbose:
|
||||
print('Formatting {}'.format(filename))
|
||||
command = [binary, filename]
|
||||
if args.i:
|
||||
command.append('-i')
|
||||
if args.sort_includes:
|
||||
command.append('-sort-includes')
|
||||
command.extend(lines)
|
||||
command.extend(['-style=file', '-fallback-style=none'])
|
||||
p = subprocess.Popen(command,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=None,
|
||||
stdin=subprocess.PIPE,
|
||||
universal_newlines=True)
|
||||
stdout, stderr = p.communicate()
|
||||
if p.returncode != 0:
|
||||
sys.exit(p.returncode)
|
||||
|
||||
if not args.i:
|
||||
with open(filename, encoding="utf8") as f:
|
||||
code = f.readlines()
|
||||
formatted_code = io.StringIO(stdout).readlines()
|
||||
diff = difflib.unified_diff(code, formatted_code,
|
||||
filename, filename,
|
||||
'(before formatting)', '(after formatting)')
|
||||
diff_string = ''.join(diff)
|
||||
if len(diff_string) > 0:
|
||||
sys.stdout.write(diff_string)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
618
contrib/devtools/copyright_header.py
Executable file
618
contrib/devtools/copyright_header.py
Executable file
|
|
@ -0,0 +1,618 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2016-2018 The Bitcoin Core developers
|
||||
# Copyright (c) 2016-2020 The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
import re
|
||||
import fnmatch
|
||||
import sys
|
||||
import subprocess
|
||||
import datetime
|
||||
import os
|
||||
|
||||
################################################################################
|
||||
# file filtering
|
||||
################################################################################
|
||||
|
||||
EXCLUDE = [
|
||||
# auto generated:
|
||||
'src/qt/digibytestrings.cpp',
|
||||
'src/chainparamsseeds.h',
|
||||
# other external copyrights:
|
||||
'src/reverse_iterator.h',
|
||||
'src/test/fuzz/FuzzedDataProvider.h',
|
||||
'src/tinyformat.h',
|
||||
'src/bench/nanobench.h',
|
||||
'test/functional/test_framework/bignum.py',
|
||||
# python init:
|
||||
'*__init__.py',
|
||||
]
|
||||
EXCLUDE_COMPILED = re.compile('|'.join([fnmatch.translate(m) for m in EXCLUDE]))
|
||||
|
||||
EXCLUDE_DIRS = [
|
||||
# git subtrees
|
||||
"src/crypto/ctaes/",
|
||||
"src/leveldb/",
|
||||
"src/secp256k1/",
|
||||
"src/univalue/",
|
||||
"src/crc32c/",
|
||||
]
|
||||
|
||||
INCLUDE = ['*.h', '*.cpp', '*.cc', '*.c', '*.mm', '*.py', '*.sh', '*.bash-completion']
|
||||
INCLUDE_COMPILED = re.compile('|'.join([fnmatch.translate(m) for m in INCLUDE]))
|
||||
|
||||
def applies_to_file(filename):
|
||||
for excluded_dir in EXCLUDE_DIRS:
|
||||
if filename.startswith(excluded_dir):
|
||||
return False
|
||||
return ((EXCLUDE_COMPILED.match(filename) is None) and
|
||||
(INCLUDE_COMPILED.match(filename) is not None))
|
||||
|
||||
################################################################################
|
||||
# obtain list of files in repo according to INCLUDE and EXCLUDE
|
||||
################################################################################
|
||||
|
||||
GIT_LS_CMD = 'git ls-files --full-name'.split(' ')
|
||||
GIT_TOPLEVEL_CMD = 'git rev-parse --show-toplevel'.split(' ')
|
||||
|
||||
def call_git_ls(base_directory):
|
||||
out = subprocess.check_output([*GIT_LS_CMD, base_directory])
|
||||
return [f for f in out.decode("utf-8").split('\n') if f != '']
|
||||
|
||||
def call_git_toplevel():
|
||||
"Returns the absolute path to the project root"
|
||||
return subprocess.check_output(GIT_TOPLEVEL_CMD).strip().decode("utf-8")
|
||||
|
||||
def get_filenames_to_examine(base_directory):
|
||||
"Returns an array of absolute paths to any project files in the base_directory that pass the include/exclude filters"
|
||||
root = call_git_toplevel()
|
||||
filenames = call_git_ls(base_directory)
|
||||
return sorted([os.path.join(root, filename) for filename in filenames if
|
||||
applies_to_file(filename)])
|
||||
|
||||
################################################################################
|
||||
# define and compile regexes for the patterns we are looking for
|
||||
################################################################################
|
||||
|
||||
|
||||
COPYRIGHT_WITH_C = r'Copyright \(c\)'
|
||||
COPYRIGHT_WITHOUT_C = 'Copyright'
|
||||
ANY_COPYRIGHT_STYLE = '(%s|%s)' % (COPYRIGHT_WITH_C, COPYRIGHT_WITHOUT_C)
|
||||
|
||||
YEAR = "20[0-9][0-9]"
|
||||
YEAR_RANGE = '(%s)(-%s)?' % (YEAR, YEAR)
|
||||
YEAR_LIST = '(%s)(, %s)+' % (YEAR, YEAR)
|
||||
ANY_YEAR_STYLE = '(%s|%s)' % (YEAR_RANGE, YEAR_LIST)
|
||||
ANY_COPYRIGHT_STYLE_OR_YEAR_STYLE = ("%s %s" % (ANY_COPYRIGHT_STYLE,
|
||||
ANY_YEAR_STYLE))
|
||||
|
||||
ANY_COPYRIGHT_COMPILED = re.compile(ANY_COPYRIGHT_STYLE_OR_YEAR_STYLE)
|
||||
|
||||
def compile_copyright_regex(copyright_style, year_style, name):
|
||||
return re.compile(r'%s %s,? %s( +\*)?\n' % (copyright_style, year_style, name))
|
||||
|
||||
EXPECTED_HOLDER_NAMES = [
|
||||
"Satoshi Nakamoto\n",
|
||||
"The DigiByte Core developers\n",
|
||||
"The DigiByte Core developers \n",
|
||||
"DigiByte Core Developers\n",
|
||||
"the DigiByte Core developers\n",
|
||||
"The DigiByte developers\n",
|
||||
"The LevelDB Authors\. All rights reserved\.\n",
|
||||
"BitPay Inc\.\n",
|
||||
"BitPay, Inc\.\n",
|
||||
"University of Illinois at Urbana-Champaign\.\n",
|
||||
"MarcoFalke\n",
|
||||
"Pieter Wuille\n",
|
||||
"Pieter Wuille +\*\n",
|
||||
"Pieter Wuille, Gregory Maxwell +\*\n",
|
||||
"Pieter Wuille, Andrew Poelstra +\*\n",
|
||||
"Andrew Poelstra +\*\n",
|
||||
"Wladimir J. van der Laan\n",
|
||||
"Jeff Garzik\n",
|
||||
"Diederik Huys, Pieter Wuille +\*\n",
|
||||
"Thomas Daede, Cory Fields +\*\n",
|
||||
"Jan-Klaas Kollhof\n",
|
||||
"Sam Rushing\n",
|
||||
"ArtForz -- public domain half-a-node\n",
|
||||
]
|
||||
|
||||
DOMINANT_STYLE_COMPILED = {}
|
||||
YEAR_LIST_STYLE_COMPILED = {}
|
||||
WITHOUT_C_STYLE_COMPILED = {}
|
||||
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
DOMINANT_STYLE_COMPILED[holder_name] = (
|
||||
compile_copyright_regex(COPYRIGHT_WITH_C, YEAR_RANGE, holder_name))
|
||||
YEAR_LIST_STYLE_COMPILED[holder_name] = (
|
||||
compile_copyright_regex(COPYRIGHT_WITH_C, YEAR_LIST, holder_name))
|
||||
WITHOUT_C_STYLE_COMPILED[holder_name] = (
|
||||
compile_copyright_regex(COPYRIGHT_WITHOUT_C, ANY_YEAR_STYLE,
|
||||
holder_name))
|
||||
|
||||
################################################################################
|
||||
# search file contents for copyright message of particular category
|
||||
################################################################################
|
||||
|
||||
def get_count_of_copyrights_of_any_style_any_holder(contents):
|
||||
return len(ANY_COPYRIGHT_COMPILED.findall(contents))
|
||||
|
||||
def file_has_dominant_style_copyright_for_holder(contents, holder_name):
|
||||
match = DOMINANT_STYLE_COMPILED[holder_name].search(contents)
|
||||
return match is not None
|
||||
|
||||
def file_has_year_list_style_copyright_for_holder(contents, holder_name):
|
||||
match = YEAR_LIST_STYLE_COMPILED[holder_name].search(contents)
|
||||
return match is not None
|
||||
|
||||
def file_has_without_c_style_copyright_for_holder(contents, holder_name):
|
||||
match = WITHOUT_C_STYLE_COMPILED[holder_name].search(contents)
|
||||
return match is not None
|
||||
|
||||
################################################################################
|
||||
# get file info
|
||||
################################################################################
|
||||
|
||||
def read_file(filename):
|
||||
return open(filename, 'r', encoding="utf8").read()
|
||||
|
||||
def gather_file_info(filename):
|
||||
info = {}
|
||||
info['filename'] = filename
|
||||
c = read_file(filename)
|
||||
info['contents'] = c
|
||||
|
||||
info['all_copyrights'] = get_count_of_copyrights_of_any_style_any_holder(c)
|
||||
|
||||
info['classified_copyrights'] = 0
|
||||
info['dominant_style'] = {}
|
||||
info['year_list_style'] = {}
|
||||
info['without_c_style'] = {}
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
has_dominant_style = (
|
||||
file_has_dominant_style_copyright_for_holder(c, holder_name))
|
||||
has_year_list_style = (
|
||||
file_has_year_list_style_copyright_for_holder(c, holder_name))
|
||||
has_without_c_style = (
|
||||
file_has_without_c_style_copyright_for_holder(c, holder_name))
|
||||
info['dominant_style'][holder_name] = has_dominant_style
|
||||
info['year_list_style'][holder_name] = has_year_list_style
|
||||
info['without_c_style'][holder_name] = has_without_c_style
|
||||
if has_dominant_style or has_year_list_style or has_without_c_style:
|
||||
info['classified_copyrights'] = info['classified_copyrights'] + 1
|
||||
return info
|
||||
|
||||
################################################################################
|
||||
# report execution
|
||||
################################################################################
|
||||
|
||||
SEPARATOR = '-'.join(['' for _ in range(80)])
|
||||
|
||||
def print_filenames(filenames, verbose):
|
||||
if not verbose:
|
||||
return
|
||||
for filename in filenames:
|
||||
print("\t%s" % filename)
|
||||
|
||||
def print_report(file_infos, verbose):
|
||||
print(SEPARATOR)
|
||||
examined = [i['filename'] for i in file_infos]
|
||||
print("%d files examined according to INCLUDE and EXCLUDE fnmatch rules" %
|
||||
len(examined))
|
||||
print_filenames(examined, verbose)
|
||||
|
||||
print(SEPARATOR)
|
||||
print('')
|
||||
zero_copyrights = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] == 0]
|
||||
print("%4d with zero copyrights" % len(zero_copyrights))
|
||||
print_filenames(zero_copyrights, verbose)
|
||||
one_copyright = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] == 1]
|
||||
print("%4d with one copyright" % len(one_copyright))
|
||||
print_filenames(one_copyright, verbose)
|
||||
two_copyrights = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] == 2]
|
||||
print("%4d with two copyrights" % len(two_copyrights))
|
||||
print_filenames(two_copyrights, verbose)
|
||||
three_copyrights = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] == 3]
|
||||
print("%4d with three copyrights" % len(three_copyrights))
|
||||
print_filenames(three_copyrights, verbose)
|
||||
four_or_more_copyrights = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] >= 4]
|
||||
print("%4d with four or more copyrights" % len(four_or_more_copyrights))
|
||||
print_filenames(four_or_more_copyrights, verbose)
|
||||
print('')
|
||||
print(SEPARATOR)
|
||||
print('Copyrights with dominant style:\ne.g. "Copyright (c)" and '
|
||||
'"<year>" or "<startYear>-<endYear>":\n')
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
dominant_style = [i['filename'] for i in file_infos if
|
||||
i['dominant_style'][holder_name]]
|
||||
if len(dominant_style) > 0:
|
||||
print("%4d with '%s'" % (len(dominant_style),
|
||||
holder_name.replace('\n', '\\n')))
|
||||
print_filenames(dominant_style, verbose)
|
||||
print('')
|
||||
print(SEPARATOR)
|
||||
print('Copyrights with year list style:\ne.g. "Copyright (c)" and '
|
||||
'"<year1>, <year2>, ...":\n')
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
year_list_style = [i['filename'] for i in file_infos if
|
||||
i['year_list_style'][holder_name]]
|
||||
if len(year_list_style) > 0:
|
||||
print("%4d with '%s'" % (len(year_list_style),
|
||||
holder_name.replace('\n', '\\n')))
|
||||
print_filenames(year_list_style, verbose)
|
||||
print('')
|
||||
print(SEPARATOR)
|
||||
print('Copyrights with no "(c)" style:\ne.g. "Copyright" and "<year>" or '
|
||||
'"<startYear>-<endYear>":\n')
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
without_c_style = [i['filename'] for i in file_infos if
|
||||
i['without_c_style'][holder_name]]
|
||||
if len(without_c_style) > 0:
|
||||
print("%4d with '%s'" % (len(without_c_style),
|
||||
holder_name.replace('\n', '\\n')))
|
||||
print_filenames(without_c_style, verbose)
|
||||
|
||||
print('')
|
||||
print(SEPARATOR)
|
||||
|
||||
unclassified_copyrights = [i['filename'] for i in file_infos if
|
||||
i['classified_copyrights'] < i['all_copyrights']]
|
||||
print("%d with unexpected copyright holder names" %
|
||||
len(unclassified_copyrights))
|
||||
print_filenames(unclassified_copyrights, verbose)
|
||||
print(SEPARATOR)
|
||||
|
||||
def exec_report(base_directory, verbose):
|
||||
filenames = get_filenames_to_examine(base_directory)
|
||||
file_infos = [gather_file_info(f) for f in filenames]
|
||||
print_report(file_infos, verbose)
|
||||
|
||||
################################################################################
|
||||
# report cmd
|
||||
################################################################################
|
||||
|
||||
REPORT_USAGE = """
|
||||
Produces a report of all copyright header notices found inside the source files
|
||||
of a repository.
|
||||
|
||||
Usage:
|
||||
$ ./copyright_header.py report <base_directory> [verbose]
|
||||
|
||||
Arguments:
|
||||
<base_directory> - The base directory of a digibyte source code repository.
|
||||
[verbose] - Includes a list of every file of each subcategory in the report.
|
||||
"""
|
||||
|
||||
def report_cmd(argv):
|
||||
if len(argv) == 2:
|
||||
sys.exit(REPORT_USAGE)
|
||||
|
||||
base_directory = argv[2]
|
||||
if not os.path.exists(base_directory):
|
||||
sys.exit("*** bad <base_directory>: %s" % base_directory)
|
||||
|
||||
if len(argv) == 3:
|
||||
verbose = False
|
||||
elif argv[3] == 'verbose':
|
||||
verbose = True
|
||||
else:
|
||||
sys.exit("*** unknown argument: %s" % argv[2])
|
||||
|
||||
exec_report(base_directory, verbose)
|
||||
|
||||
################################################################################
|
||||
# query git for year of last change
|
||||
################################################################################
|
||||
|
||||
GIT_LOG_CMD = "git log --pretty=format:%%ai %s"
|
||||
|
||||
def call_git_log(filename):
|
||||
out = subprocess.check_output((GIT_LOG_CMD % filename).split(' '))
|
||||
return out.decode("utf-8").split('\n')
|
||||
|
||||
def get_git_change_years(filename):
|
||||
git_log_lines = call_git_log(filename)
|
||||
if len(git_log_lines) == 0:
|
||||
return [datetime.date.today().year]
|
||||
# timestamp is in ISO 8601 format. e.g. "2016-09-05 14:25:32 -0600"
|
||||
return [line.split(' ')[0].split('-')[0] for line in git_log_lines]
|
||||
|
||||
def get_most_recent_git_change_year(filename):
|
||||
return max(get_git_change_years(filename))
|
||||
|
||||
################################################################################
|
||||
# read and write to file
|
||||
################################################################################
|
||||
|
||||
def read_file_lines(filename):
|
||||
f = open(filename, 'r', encoding="utf8")
|
||||
file_lines = f.readlines()
|
||||
f.close()
|
||||
return file_lines
|
||||
|
||||
def write_file_lines(filename, file_lines):
|
||||
f = open(filename, 'w', encoding="utf8")
|
||||
f.write(''.join(file_lines))
|
||||
f.close()
|
||||
|
||||
################################################################################
|
||||
# update header years execution
|
||||
################################################################################
|
||||
|
||||
COPYRIGHT = r'Copyright \(c\)'
|
||||
YEAR = "20[0-9][0-9]"
|
||||
YEAR_RANGE = '(%s)(-%s)?' % (YEAR, YEAR)
|
||||
HOLDER = 'The DigiByte Core developers'
|
||||
UPDATEABLE_LINE_COMPILED = re.compile(' '.join([COPYRIGHT, YEAR_RANGE, HOLDER]))
|
||||
|
||||
def get_updatable_copyright_line(file_lines):
|
||||
index = 0
|
||||
for line in file_lines:
|
||||
if UPDATEABLE_LINE_COMPILED.search(line) is not None:
|
||||
return index, line
|
||||
index = index + 1
|
||||
return None, None
|
||||
|
||||
def parse_year_range(year_range):
|
||||
year_split = year_range.split('-')
|
||||
start_year = year_split[0]
|
||||
if len(year_split) == 1:
|
||||
return start_year, start_year
|
||||
return start_year, year_split[1]
|
||||
|
||||
def year_range_to_str(start_year, end_year):
|
||||
if start_year == end_year:
|
||||
return start_year
|
||||
return "%s-%s" % (start_year, end_year)
|
||||
|
||||
def create_updated_copyright_line(line, last_git_change_year):
|
||||
copyright_splitter = 'Copyright (c) '
|
||||
copyright_split = line.split(copyright_splitter)
|
||||
# Preserve characters on line that are ahead of the start of the copyright
|
||||
# notice - they are part of the comment block and vary from file-to-file.
|
||||
before_copyright = copyright_split[0]
|
||||
after_copyright = copyright_split[1]
|
||||
|
||||
space_split = after_copyright.split(' ')
|
||||
year_range = space_split[0]
|
||||
start_year, end_year = parse_year_range(year_range)
|
||||
if end_year == last_git_change_year:
|
||||
return line
|
||||
return (before_copyright + copyright_splitter +
|
||||
year_range_to_str(start_year, last_git_change_year) + ' ' +
|
||||
' '.join(space_split[1:]))
|
||||
|
||||
def update_updatable_copyright(filename):
|
||||
file_lines = read_file_lines(filename)
|
||||
index, line = get_updatable_copyright_line(file_lines)
|
||||
if not line:
|
||||
print_file_action_message(filename, "No updatable copyright.")
|
||||
return
|
||||
last_git_change_year = get_most_recent_git_change_year(filename)
|
||||
new_line = create_updated_copyright_line(line, last_git_change_year)
|
||||
if line == new_line:
|
||||
print_file_action_message(filename, "Copyright up-to-date.")
|
||||
return
|
||||
file_lines[index] = new_line
|
||||
write_file_lines(filename, file_lines)
|
||||
print_file_action_message(filename,
|
||||
"Copyright updated! -> %s" % last_git_change_year)
|
||||
|
||||
def exec_update_header_year(base_directory):
|
||||
for filename in get_filenames_to_examine(base_directory):
|
||||
update_updatable_copyright(filename)
|
||||
|
||||
################################################################################
|
||||
# update cmd
|
||||
################################################################################
|
||||
|
||||
UPDATE_USAGE = """
|
||||
Updates all the copyright headers of "The DigiByte Core developers" which were
|
||||
changed in a year more recent than is listed. For example:
|
||||
|
||||
// Copyright (c) <firstYear>-<lastYear> The DigiByte Core developers
|
||||
|
||||
will be updated to:
|
||||
|
||||
// Copyright (c) <firstYear>-<lastModifiedYear> The DigiByte Core developers
|
||||
|
||||
where <lastModifiedYear> is obtained from the 'git log' history.
|
||||
|
||||
This subcommand also handles copyright headers that have only a single year. In those cases:
|
||||
|
||||
// Copyright (c) <year> The DigiByte Core developers
|
||||
|
||||
will be updated to:
|
||||
|
||||
// Copyright (c) <year>-<lastModifiedYear> The DigiByte Core developers
|
||||
|
||||
where the update is appropriate.
|
||||
|
||||
Usage:
|
||||
$ ./copyright_header.py update <base_directory>
|
||||
|
||||
Arguments:
|
||||
<base_directory> - The base directory of a digibyte source code repository.
|
||||
"""
|
||||
|
||||
def print_file_action_message(filename, action):
|
||||
print("%-52s %s" % (filename, action))
|
||||
|
||||
def update_cmd(argv):
|
||||
if len(argv) != 3:
|
||||
sys.exit(UPDATE_USAGE)
|
||||
|
||||
base_directory = argv[2]
|
||||
if not os.path.exists(base_directory):
|
||||
sys.exit("*** bad base_directory: %s" % base_directory)
|
||||
exec_update_header_year(base_directory)
|
||||
|
||||
################################################################################
|
||||
# inserted copyright header format
|
||||
################################################################################
|
||||
|
||||
def get_header_lines(header, start_year, end_year):
|
||||
lines = header.split('\n')[1:-1]
|
||||
lines[0] = lines[0] % year_range_to_str(start_year, end_year)
|
||||
return [line + '\n' for line in lines]
|
||||
|
||||
CPP_HEADER = '''
|
||||
// Copyright (c) %s The DigiByte Core developers
|
||||
// Distributed under the MIT software license, see the accompanying
|
||||
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
|
||||
def get_cpp_header_lines_to_insert(start_year, end_year):
|
||||
return reversed(get_header_lines(CPP_HEADER, start_year, end_year))
|
||||
|
||||
SCRIPT_HEADER = '''
|
||||
# Copyright (c) %s The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
|
||||
def get_script_header_lines_to_insert(start_year, end_year):
|
||||
return reversed(get_header_lines(SCRIPT_HEADER, start_year, end_year))
|
||||
|
||||
################################################################################
|
||||
# query git for year of last change
|
||||
################################################################################
|
||||
|
||||
def get_git_change_year_range(filename):
|
||||
years = get_git_change_years(filename)
|
||||
return min(years), max(years)
|
||||
|
||||
################################################################################
|
||||
# check for existing core copyright
|
||||
################################################################################
|
||||
|
||||
def file_already_has_core_copyright(file_lines):
|
||||
index, _ = get_updatable_copyright_line(file_lines)
|
||||
return index is not None
|
||||
|
||||
################################################################################
|
||||
# insert header execution
|
||||
################################################################################
|
||||
|
||||
def file_has_hashbang(file_lines):
|
||||
if len(file_lines) < 1:
|
||||
return False
|
||||
if len(file_lines[0]) <= 2:
|
||||
return False
|
||||
return file_lines[0][:2] == '#!'
|
||||
|
||||
def insert_script_header(filename, file_lines, start_year, end_year):
|
||||
if file_has_hashbang(file_lines):
|
||||
insert_idx = 1
|
||||
else:
|
||||
insert_idx = 0
|
||||
header_lines = get_script_header_lines_to_insert(start_year, end_year)
|
||||
for line in header_lines:
|
||||
file_lines.insert(insert_idx, line)
|
||||
write_file_lines(filename, file_lines)
|
||||
|
||||
def insert_cpp_header(filename, file_lines, start_year, end_year):
|
||||
file_lines.insert(0, '\n')
|
||||
header_lines = get_cpp_header_lines_to_insert(start_year, end_year)
|
||||
for line in header_lines:
|
||||
file_lines.insert(0, line)
|
||||
write_file_lines(filename, file_lines)
|
||||
|
||||
def exec_insert_header(filename, style):
|
||||
file_lines = read_file_lines(filename)
|
||||
if file_already_has_core_copyright(file_lines):
|
||||
sys.exit('*** %s already has a copyright by The DigiByte Core developers'
|
||||
% (filename))
|
||||
start_year, end_year = get_git_change_year_range(filename)
|
||||
if style in ['python', 'shell']:
|
||||
insert_script_header(filename, file_lines, start_year, end_year)
|
||||
else:
|
||||
insert_cpp_header(filename, file_lines, start_year, end_year)
|
||||
|
||||
################################################################################
|
||||
# insert cmd
|
||||
################################################################################
|
||||
|
||||
INSERT_USAGE = """
|
||||
Inserts a copyright header for "The DigiByte Core developers" at the top of the
|
||||
file in either Python or C++ style as determined by the file extension. If the
|
||||
file is a Python file and it has a '#!' starting the first line, the header is
|
||||
inserted in the line below it.
|
||||
|
||||
The copyright dates will be set to be:
|
||||
|
||||
"<year_introduced>-<current_year>"
|
||||
|
||||
where <year_introduced> is according to the 'git log' history. If
|
||||
<year_introduced> is equal to <current_year>, the date will be set to be:
|
||||
|
||||
"<current_year>"
|
||||
|
||||
If the file already has a copyright for "The DigiByte Core developers", the
|
||||
script will exit.
|
||||
|
||||
Usage:
|
||||
$ ./copyright_header.py insert <file>
|
||||
|
||||
Arguments:
|
||||
<file> - A source file in the digibyte repository.
|
||||
"""
|
||||
|
||||
def insert_cmd(argv):
|
||||
if len(argv) != 3:
|
||||
sys.exit(INSERT_USAGE)
|
||||
|
||||
filename = argv[2]
|
||||
if not os.path.isfile(filename):
|
||||
sys.exit("*** bad filename: %s" % filename)
|
||||
_, extension = os.path.splitext(filename)
|
||||
if extension not in ['.h', '.cpp', '.cc', '.c', '.py', '.sh']:
|
||||
sys.exit("*** cannot insert for file extension %s" % extension)
|
||||
|
||||
if extension == '.py':
|
||||
style = 'python'
|
||||
elif extension == '.sh':
|
||||
style = 'shell'
|
||||
else:
|
||||
style = 'cpp'
|
||||
exec_insert_header(filename, style)
|
||||
|
||||
################################################################################
|
||||
# UI
|
||||
################################################################################
|
||||
|
||||
USAGE = """
|
||||
copyright_header.py - utilities for managing copyright headers of 'The DigiByte
|
||||
Core developers' in repository source files.
|
||||
|
||||
Usage:
|
||||
$ ./copyright_header <subcommand>
|
||||
|
||||
Subcommands:
|
||||
report
|
||||
update
|
||||
insert
|
||||
|
||||
To see subcommand usage, run them without arguments.
|
||||
"""
|
||||
|
||||
SUBCOMMANDS = ['report', 'update', 'insert']
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) == 1:
|
||||
sys.exit(USAGE)
|
||||
subcommand = sys.argv[1]
|
||||
if subcommand not in SUBCOMMANDS:
|
||||
sys.exit(USAGE)
|
||||
if subcommand == 'report':
|
||||
report_cmd(sys.argv)
|
||||
elif subcommand == 'update':
|
||||
update_cmd(sys.argv)
|
||||
elif subcommand == 'insert':
|
||||
insert_cmd(sys.argv)
|
||||
53
contrib/devtools/gen-manpages.sh
Executable file
53
contrib/devtools/gen-manpages.sh
Executable file
|
|
@ -0,0 +1,53 @@
|
|||
#!/usr/bin/env bash
|
||||
# Copyright (c) 2016-2020 The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
export LC_ALL=C
|
||||
TOPDIR=${TOPDIR:-$(git rev-parse --show-toplevel)}
|
||||
BUILDDIR=${BUILDDIR:-$TOPDIR}
|
||||
|
||||
BINDIR=${BINDIR:-$BUILDDIR/src}
|
||||
MANDIR=${MANDIR:-$TOPDIR/doc/man}
|
||||
|
||||
DIGIBYTED=${DIGIBYTED:-$BINDIR/digibyted}
|
||||
DIGIBYTECLI=${DIGIBYTECLI:-$BINDIR/digibyte-cli}
|
||||
DIGIBYTETX=${DIGIBYTETX:-$BINDIR/digibyte-tx}
|
||||
WALLET_TOOL=${WALLET_TOOL:-$BINDIR/digibyte-wallet}
|
||||
DIGIBYTEUTIL=${DIGIBYTEQT:-$BINDIR/digibyte-util}
|
||||
DIGIBYTEQT=${DIGIBYTEQT:-$BINDIR/qt/digibyte-qt}
|
||||
|
||||
[ ! -x $DIGIBYTED ] && echo "$DIGIBYTED not found or not executable." && exit 1
|
||||
|
||||
# Don't allow man pages to be generated for binaries built from a dirty tree
|
||||
DIRTY=""
|
||||
for cmd in $DIGIBYTED $DIGIBYTECLI $DIGIBYTETX $WALLET_TOOL $DIGIBYTEUTIL $DIGIBYTEQT; do
|
||||
VERSION_OUTPUT=$($cmd --version)
|
||||
if [[ $VERSION_OUTPUT == *"dirty"* ]]; then
|
||||
DIRTY="${DIRTY}${cmd}\n"
|
||||
fi
|
||||
done
|
||||
if [ -n "$DIRTY" ]
|
||||
then
|
||||
echo -e "WARNING: the following binaries were built from a dirty tree:\n"
|
||||
echo -e $DIRTY
|
||||
echo "man pages generated from dirty binaries should NOT be committed."
|
||||
echo "To properly generate man pages, please commit your changes to the above binaries, rebuild them, then run this script again."
|
||||
fi
|
||||
|
||||
# The autodetected version git tag can screw up manpage output a little bit
|
||||
read -r -a DGBVER <<< "$($DIGIBYTECLI --version | head -n1 | awk -F'[ -]' '{ print $6, $7 }')"
|
||||
|
||||
# Create a footer file with copyright content.
|
||||
# This gets autodetected fine for digibyted if --version-string is not set,
|
||||
# but has different outcomes for digibyte-qt and digibyte-cli.
|
||||
echo "[COPYRIGHT]" > footer.h2m
|
||||
$DIGIBYTED --version | sed -n '1!p' >> footer.h2m
|
||||
|
||||
for cmd in $DIGIBYTED $DIGIBYTECLI $DIGIBYTETX $WALLET_TOOL $DIGIBYTEUTIL $DIGIBYTEQT; do
|
||||
cmdname="${cmd##*/}"
|
||||
help2man -N --version-string=${DGBVER[0]} --include=footer.h2m -o ${MANDIR}/${cmdname}.1 ${cmd}
|
||||
sed -i "s/\\\-${DGBVER[1]}//g" ${MANDIR}/${cmdname}.1
|
||||
done
|
||||
|
||||
rm -f footer.h2m
|
||||
331
contrib/devtools/github-merge.py
Executable file
331
contrib/devtools/github-merge.py
Executable file
|
|
@ -0,0 +1,331 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2016-2017 DigiByte Core Developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
# This script will locally construct a merge commit for a pull request on a
|
||||
# github repository, inspect it, sign it and optionally push it.
|
||||
|
||||
# The following temporary branches are created/overwritten and deleted:
|
||||
# * pull/$PULL/base (the current master we're merging onto)
|
||||
# * pull/$PULL/head (the current state of the remote pull request)
|
||||
# * pull/$PULL/merge (github's merge)
|
||||
# * pull/$PULL/local-merge (our merge)
|
||||
|
||||
# In case of a clean merge that is accepted by the user, the local branch with
|
||||
# name $BRANCH is overwritten with the merged result, and optionally pushed.
|
||||
from __future__ import division,print_function,unicode_literals
|
||||
import os
|
||||
from sys import stdin,stdout,stderr
|
||||
import argparse
|
||||
import hashlib
|
||||
import subprocess
|
||||
import sys
|
||||
import json
|
||||
import codecs
|
||||
try:
|
||||
from urllib.request import Request,urlopen
|
||||
except:
|
||||
from urllib2 import Request,urlopen
|
||||
|
||||
# External tools (can be overridden using environment)
|
||||
GIT = os.getenv('GIT','git')
|
||||
BASH = os.getenv('BASH','bash')
|
||||
|
||||
# OS specific configuration for terminal attributes
|
||||
ATTR_RESET = ''
|
||||
ATTR_PR = ''
|
||||
COMMIT_FORMAT = '%h %s (%an)%d'
|
||||
if os.name == 'posix': # if posix, assume we can use basic terminal escapes
|
||||
ATTR_RESET = '\033[0m'
|
||||
ATTR_PR = '\033[1;36m'
|
||||
COMMIT_FORMAT = '%C(bold blue)%h%Creset %s %C(cyan)(%an)%Creset%C(green)%d%Creset'
|
||||
|
||||
def git_config_get(option, default=None):
|
||||
'''
|
||||
Get named configuration option from git repository.
|
||||
'''
|
||||
try:
|
||||
return subprocess.check_output([GIT,'config','--get',option]).rstrip().decode('utf-8')
|
||||
except subprocess.CalledProcessError:
|
||||
return default
|
||||
|
||||
def retrieve_pr_info(repo,pull):
|
||||
'''
|
||||
Retrieve pull request information from github.
|
||||
Return None if no title can be found, or an error happens.
|
||||
'''
|
||||
try:
|
||||
req = Request("https://api.github.com/repos/"+repo+"/pulls/"+pull)
|
||||
result = urlopen(req)
|
||||
reader = codecs.getreader('utf-8')
|
||||
obj = json.load(reader(result))
|
||||
return obj
|
||||
except Exception as e:
|
||||
print('Warning: unable to retrieve pull information from github: %s' % e)
|
||||
return None
|
||||
|
||||
def ask_prompt(text):
|
||||
print(text,end=" ",file=stderr)
|
||||
stderr.flush()
|
||||
reply = stdin.readline().rstrip()
|
||||
print("",file=stderr)
|
||||
return reply
|
||||
|
||||
def get_symlink_files():
|
||||
files = sorted(subprocess.check_output([GIT, 'ls-tree', '--full-tree', '-r', 'HEAD']).splitlines())
|
||||
ret = []
|
||||
for f in files:
|
||||
if (int(f.decode('utf-8').split(" ")[0], 8) & 0o170000) == 0o120000:
|
||||
ret.append(f.decode('utf-8').split("\t")[1])
|
||||
return ret
|
||||
|
||||
def tree_sha512sum(commit='HEAD'):
|
||||
# request metadata for entire tree, recursively
|
||||
files = []
|
||||
blob_by_name = {}
|
||||
for line in subprocess.check_output([GIT, 'ls-tree', '--full-tree', '-r', commit]).splitlines():
|
||||
name_sep = line.index(b'\t')
|
||||
metadata = line[:name_sep].split() # perms, 'blob', blobid
|
||||
assert(metadata[1] == b'blob')
|
||||
name = line[name_sep+1:]
|
||||
files.append(name)
|
||||
blob_by_name[name] = metadata[2]
|
||||
|
||||
files.sort()
|
||||
# open connection to git-cat-file in batch mode to request data for all blobs
|
||||
# this is much faster than launching it per file
|
||||
p = subprocess.Popen([GIT, 'cat-file', '--batch'], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
|
||||
overall = hashlib.sha512()
|
||||
for f in files:
|
||||
blob = blob_by_name[f]
|
||||
# request blob
|
||||
p.stdin.write(blob + b'\n')
|
||||
p.stdin.flush()
|
||||
# read header: blob, "blob", size
|
||||
reply = p.stdout.readline().split()
|
||||
assert(reply[0] == blob and reply[1] == b'blob')
|
||||
size = int(reply[2])
|
||||
# hash the blob data
|
||||
intern = hashlib.sha512()
|
||||
ptr = 0
|
||||
while ptr < size:
|
||||
bs = min(65536, size - ptr)
|
||||
piece = p.stdout.read(bs)
|
||||
if len(piece) == bs:
|
||||
intern.update(piece)
|
||||
else:
|
||||
raise IOError('Premature EOF reading git cat-file output')
|
||||
ptr += bs
|
||||
dig = intern.hexdigest()
|
||||
assert(p.stdout.read(1) == b'\n') # ignore LF that follows blob data
|
||||
# update overall hash with file hash
|
||||
overall.update(dig.encode("utf-8"))
|
||||
overall.update(" ".encode("utf-8"))
|
||||
overall.update(f)
|
||||
overall.update("\n".encode("utf-8"))
|
||||
p.stdin.close()
|
||||
if p.wait():
|
||||
raise IOError('Non-zero return value executing git cat-file')
|
||||
return overall.hexdigest()
|
||||
|
||||
def print_merge_details(pull, title, branch, base_branch, head_branch):
|
||||
print('%s#%s%s %s %sinto %s%s' % (ATTR_RESET+ATTR_PR,pull,ATTR_RESET,title,ATTR_RESET+ATTR_PR,branch,ATTR_RESET))
|
||||
subprocess.check_call([GIT,'log','--graph','--topo-order','--pretty=format:'+COMMIT_FORMAT,base_branch+'..'+head_branch])
|
||||
|
||||
def parse_arguments():
|
||||
epilog = '''
|
||||
In addition, you can set the following git configuration variables:
|
||||
githubmerge.repository (mandatory),
|
||||
user.signingkey (mandatory),
|
||||
githubmerge.host (default: git@github.com),
|
||||
githubmerge.branch (no default),
|
||||
githubmerge.testcmd (default: none).
|
||||
'''
|
||||
parser = argparse.ArgumentParser(description='Utility to merge, sign and push github pull requests',
|
||||
epilog=epilog)
|
||||
parser.add_argument('pull', metavar='PULL', type=int, nargs=1,
|
||||
help='Pull request ID to merge')
|
||||
parser.add_argument('branch', metavar='BRANCH', type=str, nargs='?',
|
||||
default=None, help='Branch to merge against (default: githubmerge.branch setting, or base branch for pull, or \'master\')')
|
||||
return parser.parse_args()
|
||||
|
||||
def main():
|
||||
# Extract settings from git repo
|
||||
repo = git_config_get('githubmerge.repository')
|
||||
host = git_config_get('githubmerge.host','git@github.com')
|
||||
opt_branch = git_config_get('githubmerge.branch',None)
|
||||
testcmd = git_config_get('githubmerge.testcmd')
|
||||
signingkey = git_config_get('user.signingkey')
|
||||
if repo is None:
|
||||
print("ERROR: No repository configured. Use this command to set:", file=stderr)
|
||||
print("git config githubmerge.repository <owner>/<repo>", file=stderr)
|
||||
sys.exit(1)
|
||||
if signingkey is None:
|
||||
print("ERROR: No GPG signing key set. Set one using:",file=stderr)
|
||||
print("git config --global user.signingkey <key>",file=stderr)
|
||||
sys.exit(1)
|
||||
|
||||
host_repo = host+":"+repo # shortcut for push/pull target
|
||||
|
||||
# Extract settings from command line
|
||||
args = parse_arguments()
|
||||
pull = str(args.pull[0])
|
||||
|
||||
# Receive pull information from github
|
||||
info = retrieve_pr_info(repo,pull)
|
||||
if info is None:
|
||||
sys.exit(1)
|
||||
title = info['title'].strip()
|
||||
body = info['body'].strip()
|
||||
# precedence order for destination branch argument:
|
||||
# - command line argument
|
||||
# - githubmerge.branch setting
|
||||
# - base branch for pull (as retrieved from github)
|
||||
# - 'master'
|
||||
branch = args.branch or opt_branch or info['base']['ref'] or 'master'
|
||||
|
||||
# Initialize source branches
|
||||
head_branch = 'pull/'+pull+'/head'
|
||||
base_branch = 'pull/'+pull+'/base'
|
||||
merge_branch = 'pull/'+pull+'/merge'
|
||||
local_merge_branch = 'pull/'+pull+'/local-merge'
|
||||
|
||||
devnull = open(os.devnull, 'w', encoding="utf8")
|
||||
try:
|
||||
subprocess.check_call([GIT,'checkout','-q',branch])
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot check out branch %s." % (branch), file=stderr)
|
||||
sys.exit(3)
|
||||
try:
|
||||
subprocess.check_call([GIT,'fetch','-q',host_repo,'+refs/pull/'+pull+'/*:refs/heads/pull/'+pull+'/*',
|
||||
'+refs/heads/'+branch+':refs/heads/'+base_branch])
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot find pull request #%s or branch %s on %s." % (pull,branch,host_repo), file=stderr)
|
||||
sys.exit(3)
|
||||
try:
|
||||
subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+head_branch], stdout=devnull, stderr=stdout)
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot find head of pull request #%s on %s." % (pull,host_repo), file=stderr)
|
||||
sys.exit(3)
|
||||
try:
|
||||
subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+merge_branch], stdout=devnull, stderr=stdout)
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot find merge of pull request #%s on %s." % (pull,host_repo), file=stderr)
|
||||
sys.exit(3)
|
||||
subprocess.check_call([GIT,'checkout','-q',base_branch])
|
||||
subprocess.call([GIT,'branch','-q','-D',local_merge_branch], stderr=devnull)
|
||||
subprocess.check_call([GIT,'checkout','-q','-b',local_merge_branch])
|
||||
|
||||
try:
|
||||
# Go up to the repository's root.
|
||||
toplevel = subprocess.check_output([GIT,'rev-parse','--show-toplevel']).strip()
|
||||
os.chdir(toplevel)
|
||||
# Create unsigned merge commit.
|
||||
if title:
|
||||
firstline = 'Merge #%s: %s' % (pull,title)
|
||||
else:
|
||||
firstline = 'Merge #%s' % (pull,)
|
||||
message = firstline + '\n\n'
|
||||
message += subprocess.check_output([GIT,'log','--no-merges','--topo-order','--pretty=format:%h %s (%an)',base_branch+'..'+head_branch]).decode('utf-8')
|
||||
message += '\n\nPull request description:\n\n ' + body.replace('\n', '\n ') + '\n'
|
||||
try:
|
||||
subprocess.check_call([GIT,'merge','-q','--commit','--no-edit','--no-ff','-m',message.encode('utf-8'),head_branch])
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot be merged cleanly.",file=stderr)
|
||||
subprocess.check_call([GIT,'merge','--abort'])
|
||||
sys.exit(4)
|
||||
logmsg = subprocess.check_output([GIT,'log','--pretty=format:%s','-n','1']).decode('utf-8')
|
||||
if logmsg.rstrip() != firstline.rstrip():
|
||||
print("ERROR: Creating merge failed (already merged?).",file=stderr)
|
||||
sys.exit(4)
|
||||
|
||||
symlink_files = get_symlink_files()
|
||||
for f in symlink_files:
|
||||
print("ERROR: File %s was a symlink" % f)
|
||||
if len(symlink_files) > 0:
|
||||
sys.exit(4)
|
||||
|
||||
# Put tree SHA512 into the message
|
||||
try:
|
||||
first_sha512 = tree_sha512sum()
|
||||
message += '\n\nTree-SHA512: ' + first_sha512
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Unable to compute tree hash")
|
||||
sys.exit(4)
|
||||
try:
|
||||
subprocess.check_call([GIT,'commit','--amend','-m',message.encode('utf-8')])
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot update message.", file=stderr)
|
||||
sys.exit(4)
|
||||
|
||||
print_merge_details(pull, title, branch, base_branch, head_branch)
|
||||
print()
|
||||
|
||||
# Run test command if configured.
|
||||
if testcmd:
|
||||
if subprocess.call(testcmd,shell=True):
|
||||
print("ERROR: Running %s failed." % testcmd,file=stderr)
|
||||
sys.exit(5)
|
||||
|
||||
# Show the created merge.
|
||||
diff = subprocess.check_output([GIT,'diff',merge_branch+'..'+local_merge_branch])
|
||||
subprocess.check_call([GIT,'diff',base_branch+'..'+local_merge_branch])
|
||||
if diff:
|
||||
print("WARNING: merge differs from github!",file=stderr)
|
||||
reply = ask_prompt("Type 'ignore' to continue.")
|
||||
if reply.lower() == 'ignore':
|
||||
print("Difference with github ignored.",file=stderr)
|
||||
else:
|
||||
sys.exit(6)
|
||||
else:
|
||||
# Verify the result manually.
|
||||
print("Dropping you on a shell so you can try building/testing the merged source.",file=stderr)
|
||||
print("Run 'git diff HEAD~' to show the changes being merged.",file=stderr)
|
||||
print("Type 'exit' when done.",file=stderr)
|
||||
if os.path.isfile('/etc/debian_version'): # Show pull number on Debian default prompt
|
||||
os.putenv('debian_chroot',pull)
|
||||
subprocess.call([BASH,'-i'])
|
||||
|
||||
second_sha512 = tree_sha512sum()
|
||||
if first_sha512 != second_sha512:
|
||||
print("ERROR: Tree hash changed unexpectedly",file=stderr)
|
||||
sys.exit(8)
|
||||
|
||||
# Sign the merge commit.
|
||||
print_merge_details(pull, title, branch, base_branch, head_branch)
|
||||
while True:
|
||||
reply = ask_prompt("Type 's' to sign off on the above merge, or 'x' to reject and exit.").lower()
|
||||
if reply == 's':
|
||||
try:
|
||||
subprocess.check_call([GIT,'commit','-q','--gpg-sign','--amend','--no-edit'])
|
||||
break
|
||||
except subprocess.CalledProcessError:
|
||||
print("Error while signing, asking again.",file=stderr)
|
||||
elif reply == 'x':
|
||||
print("Not signing off on merge, exiting.",file=stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Put the result in branch.
|
||||
subprocess.check_call([GIT,'checkout','-q',branch])
|
||||
subprocess.check_call([GIT,'reset','-q','--hard',local_merge_branch])
|
||||
finally:
|
||||
# Clean up temporary branches.
|
||||
subprocess.call([GIT,'checkout','-q',branch])
|
||||
subprocess.call([GIT,'branch','-q','-D',head_branch],stderr=devnull)
|
||||
subprocess.call([GIT,'branch','-q','-D',base_branch],stderr=devnull)
|
||||
subprocess.call([GIT,'branch','-q','-D',merge_branch],stderr=devnull)
|
||||
subprocess.call([GIT,'branch','-q','-D',local_merge_branch],stderr=devnull)
|
||||
|
||||
# Push the result.
|
||||
while True:
|
||||
reply = ask_prompt("Type 'push' to push the result to %s, branch %s, or 'x' to exit without pushing." % (host_repo,branch)).lower()
|
||||
if reply == 'push':
|
||||
subprocess.check_call([GIT,'push',host_repo,'refs/heads/'+branch])
|
||||
break
|
||||
elif reply == 'x':
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
76
contrib/devtools/optimize-pngs.py
Executable file
76
contrib/devtools/optimize-pngs.py
Executable file
|
|
@ -0,0 +1,76 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2014-2018 The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Run this script every time you change one of the png files. Using pngcrush, it will optimize the png files, remove various color profiles, remove ancillary chunks (alla) and text chunks (text).
|
||||
#pngcrush -brute -ow -rem gAMA -rem cHRM -rem iCCP -rem sRGB -rem alla -rem text
|
||||
'''
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import hashlib
|
||||
from PIL import Image # pip3 install Pillow
|
||||
|
||||
def file_hash(filename):
|
||||
'''Return hash of raw file contents'''
|
||||
with open(filename, 'rb') as f:
|
||||
return hashlib.sha256(f.read()).hexdigest()
|
||||
|
||||
def content_hash(filename):
|
||||
'''Return hash of RGBA contents of image'''
|
||||
i = Image.open(filename)
|
||||
i = i.convert('RGBA')
|
||||
data = i.tobytes()
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
pngcrush = 'pngcrush'
|
||||
git = 'git'
|
||||
folders = ["src/qt/res/movies", "src/qt/res/icons", "share/pixmaps"]
|
||||
basePath = subprocess.check_output([git, 'rev-parse', '--show-toplevel'], universal_newlines=True).rstrip('\n')
|
||||
totalSaveBytes = 0
|
||||
noHashChange = True
|
||||
|
||||
outputArray = []
|
||||
for folder in folders:
|
||||
absFolder=os.path.join(basePath, folder)
|
||||
for file in os.listdir(absFolder):
|
||||
extension = os.path.splitext(file)[1]
|
||||
if extension.lower() == '.png':
|
||||
print("optimizing {}...".format(file), end =' ')
|
||||
file_path = os.path.join(absFolder, file)
|
||||
fileMetaMap = {'file' : file, 'osize': os.path.getsize(file_path), 'sha256Old' : file_hash(file_path)}
|
||||
fileMetaMap['contentHashPre'] = content_hash(file_path)
|
||||
|
||||
try:
|
||||
subprocess.call([pngcrush, "-brute", "-ow", "-rem", "gAMA", "-rem", "cHRM", "-rem", "iCCP", "-rem", "sRGB", "-rem", "alla", "-rem", "text", file_path],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
except:
|
||||
print("pngcrush is not installed, aborting...")
|
||||
sys.exit(0)
|
||||
|
||||
#verify
|
||||
if "Not a PNG file" in subprocess.check_output([pngcrush, "-n", "-v", file_path], stderr=subprocess.STDOUT, universal_newlines=True):
|
||||
print("PNG file "+file+" is corrupted after crushing, check out pngcursh version")
|
||||
sys.exit(1)
|
||||
|
||||
fileMetaMap['sha256New'] = file_hash(file_path)
|
||||
fileMetaMap['contentHashPost'] = content_hash(file_path)
|
||||
|
||||
if fileMetaMap['contentHashPre'] != fileMetaMap['contentHashPost']:
|
||||
print("Image contents of PNG file {} before and after crushing don't match".format(file))
|
||||
sys.exit(1)
|
||||
|
||||
fileMetaMap['psize'] = os.path.getsize(file_path)
|
||||
outputArray.append(fileMetaMap)
|
||||
print("done")
|
||||
|
||||
print("summary:\n+++++++++++++++++")
|
||||
for fileDict in outputArray:
|
||||
oldHash = fileDict['sha256Old']
|
||||
newHash = fileDict['sha256New']
|
||||
totalSaveBytes += fileDict['osize'] - fileDict['psize']
|
||||
noHashChange = noHashChange and (oldHash == newHash)
|
||||
print(fileDict['file']+"\n size diff from: "+str(fileDict['osize'])+" to: "+str(fileDict['psize'])+"\n old sha256: "+oldHash+"\n new sha256: "+newHash+"\n")
|
||||
|
||||
print("completed. Checksum stable: "+str(noHashChange)+". Total reduction: "+str(totalSaveBytes)+" bytes")
|
||||
323
contrib/devtools/pixie.py
Normal file
323
contrib/devtools/pixie.py
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2020 Wladimir J. van der Laan
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Compact, self-contained ELF implementation for digibyte-core security checks.
|
||||
'''
|
||||
import struct
|
||||
import types
|
||||
from typing import Dict, List, Optional, Union, Tuple
|
||||
|
||||
# you can find all these values in elf.h
|
||||
EI_NIDENT = 16
|
||||
|
||||
# Byte indices in e_ident
|
||||
EI_CLASS = 4 # ELFCLASSxx
|
||||
EI_DATA = 5 # ELFDATAxxxx
|
||||
|
||||
ELFCLASS32 = 1 # 32-bit
|
||||
ELFCLASS64 = 2 # 64-bit
|
||||
|
||||
ELFDATA2LSB = 1 # little endian
|
||||
ELFDATA2MSB = 2 # big endian
|
||||
|
||||
# relevant values for e_machine
|
||||
EM_386 = 3
|
||||
EM_PPC64 = 21
|
||||
EM_ARM = 40
|
||||
EM_AARCH64 = 183
|
||||
EM_X86_64 = 62
|
||||
EM_RISCV = 243
|
||||
|
||||
# relevant values for e_type
|
||||
ET_DYN = 3
|
||||
|
||||
# relevant values for sh_type
|
||||
SHT_PROGBITS = 1
|
||||
SHT_STRTAB = 3
|
||||
SHT_DYNAMIC = 6
|
||||
SHT_DYNSYM = 11
|
||||
SHT_GNU_verneed = 0x6ffffffe
|
||||
SHT_GNU_versym = 0x6fffffff
|
||||
|
||||
# relevant values for p_type
|
||||
PT_LOAD = 1
|
||||
PT_GNU_STACK = 0x6474e551
|
||||
PT_GNU_RELRO = 0x6474e552
|
||||
|
||||
# relevant values for p_flags
|
||||
PF_X = (1 << 0)
|
||||
PF_W = (1 << 1)
|
||||
PF_R = (1 << 2)
|
||||
|
||||
# relevant values for d_tag
|
||||
DT_NEEDED = 1
|
||||
DT_FLAGS = 30
|
||||
|
||||
# relevant values of `d_un.d_val' in the DT_FLAGS entry
|
||||
DF_BIND_NOW = 0x00000008
|
||||
|
||||
# relevant d_tags with string payload
|
||||
STRING_TAGS = {DT_NEEDED}
|
||||
|
||||
# rrlevant values for ST_BIND subfield of st_info (symbol binding)
|
||||
STB_LOCAL = 0
|
||||
|
||||
class ELFRecord(types.SimpleNamespace):
|
||||
'''Unified parsing for ELF records.'''
|
||||
def __init__(self, data: bytes, offset: int, eh: 'ELFHeader', total_size: Optional[int]) -> None:
|
||||
hdr_struct = self.STRUCT[eh.ei_class][0][eh.ei_data]
|
||||
if total_size is not None and hdr_struct.size > total_size:
|
||||
raise ValueError(f'{self.__class__.__name__} header size too small ({total_size} < {hdr_struct.size})')
|
||||
for field, value in zip(self.STRUCT[eh.ei_class][1], hdr_struct.unpack(data[offset:offset + hdr_struct.size])):
|
||||
setattr(self, field, value)
|
||||
|
||||
def BiStruct(chars: str) -> Dict[int, struct.Struct]:
|
||||
'''Compile a struct parser for both endians.'''
|
||||
return {
|
||||
ELFDATA2LSB: struct.Struct('<' + chars),
|
||||
ELFDATA2MSB: struct.Struct('>' + chars),
|
||||
}
|
||||
|
||||
class ELFHeader(ELFRecord):
|
||||
FIELDS = ['e_type', 'e_machine', 'e_version', 'e_entry', 'e_phoff', 'e_shoff', 'e_flags', 'e_ehsize', 'e_phentsize', 'e_phnum', 'e_shentsize', 'e_shnum', 'e_shstrndx']
|
||||
STRUCT = {
|
||||
ELFCLASS32: (BiStruct('HHIIIIIHHHHHH'), FIELDS),
|
||||
ELFCLASS64: (BiStruct('HHIQQQIHHHHHH'), FIELDS),
|
||||
}
|
||||
|
||||
def __init__(self, data: bytes, offset: int) -> None:
|
||||
self.e_ident = data[offset:offset + EI_NIDENT]
|
||||
if self.e_ident[0:4] != b'\x7fELF':
|
||||
raise ValueError('invalid ELF magic')
|
||||
self.ei_class = self.e_ident[EI_CLASS]
|
||||
self.ei_data = self.e_ident[EI_DATA]
|
||||
|
||||
super().__init__(data, offset + EI_NIDENT, self, None)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'Header(e_ident={self.e_ident!r}, e_type={self.e_type}, e_machine={self.e_machine}, e_version={self.e_version}, e_entry={self.e_entry}, e_phoff={self.e_phoff}, e_shoff={self.e_shoff}, e_flags={self.e_flags}, e_ehsize={self.e_ehsize}, e_phentsize={self.e_phentsize}, e_phnum={self.e_phnum}, e_shentsize={self.e_shentsize}, e_shnum={self.e_shnum}, e_shstrndx={self.e_shstrndx})'
|
||||
|
||||
class Section(ELFRecord):
|
||||
name: Optional[bytes] = None
|
||||
FIELDS = ['sh_name', 'sh_type', 'sh_flags', 'sh_addr', 'sh_offset', 'sh_size', 'sh_link', 'sh_info', 'sh_addralign', 'sh_entsize']
|
||||
STRUCT = {
|
||||
ELFCLASS32: (BiStruct('IIIIIIIIII'), FIELDS),
|
||||
ELFCLASS64: (BiStruct('IIQQQQIIQQ'), FIELDS),
|
||||
}
|
||||
|
||||
def __init__(self, data: bytes, offset: int, eh: ELFHeader) -> None:
|
||||
super().__init__(data, offset, eh, eh.e_shentsize)
|
||||
self._data = data
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'Section(sh_name={self.sh_name}({self.name!r}), sh_type=0x{self.sh_type:x}, sh_flags={self.sh_flags}, sh_addr=0x{self.sh_addr:x}, sh_offset=0x{self.sh_offset:x}, sh_size={self.sh_size}, sh_link={self.sh_link}, sh_info={self.sh_info}, sh_addralign={self.sh_addralign}, sh_entsize={self.sh_entsize})'
|
||||
|
||||
def contents(self) -> bytes:
|
||||
'''Return section contents.'''
|
||||
return self._data[self.sh_offset:self.sh_offset + self.sh_size]
|
||||
|
||||
class ProgramHeader(ELFRecord):
|
||||
STRUCT = {
|
||||
# different ELF classes have the same fields, but in a different order to optimize space versus alignment
|
||||
ELFCLASS32: (BiStruct('IIIIIIII'), ['p_type', 'p_offset', 'p_vaddr', 'p_paddr', 'p_filesz', 'p_memsz', 'p_flags', 'p_align']),
|
||||
ELFCLASS64: (BiStruct('IIQQQQQQ'), ['p_type', 'p_flags', 'p_offset', 'p_vaddr', 'p_paddr', 'p_filesz', 'p_memsz', 'p_align']),
|
||||
}
|
||||
|
||||
def __init__(self, data: bytes, offset: int, eh: ELFHeader) -> None:
|
||||
super().__init__(data, offset, eh, eh.e_phentsize)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'ProgramHeader(p_type={self.p_type}, p_offset={self.p_offset}, p_vaddr={self.p_vaddr}, p_paddr={self.p_paddr}, p_filesz={self.p_filesz}, p_memsz={self.p_memsz}, p_flags={self.p_flags}, p_align={self.p_align})'
|
||||
|
||||
class Symbol(ELFRecord):
|
||||
STRUCT = {
|
||||
# different ELF classes have the same fields, but in a different order to optimize space versus alignment
|
||||
ELFCLASS32: (BiStruct('IIIBBH'), ['st_name', 'st_value', 'st_size', 'st_info', 'st_other', 'st_shndx']),
|
||||
ELFCLASS64: (BiStruct('IBBHQQ'), ['st_name', 'st_info', 'st_other', 'st_shndx', 'st_value', 'st_size']),
|
||||
}
|
||||
|
||||
def __init__(self, data: bytes, offset: int, eh: ELFHeader, symtab: Section, strings: bytes, version: Optional[bytes]) -> None:
|
||||
super().__init__(data, offset, eh, symtab.sh_entsize)
|
||||
self.name = _lookup_string(strings, self.st_name)
|
||||
self.version = version
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'Symbol(st_name={self.st_name}({self.name!r}), st_value={self.st_value}, st_size={self.st_size}, st_info={self.st_info}, st_other={self.st_other}, st_shndx={self.st_shndx}, version={self.version!r})'
|
||||
|
||||
@property
|
||||
def is_import(self) -> bool:
|
||||
'''Returns whether the symbol is an imported symbol.'''
|
||||
return self.st_bind != STB_LOCAL and self.st_shndx == 0
|
||||
|
||||
@property
|
||||
def is_export(self) -> bool:
|
||||
'''Returns whether the symbol is an exported symbol.'''
|
||||
return self.st_bind != STB_LOCAL and self.st_shndx != 0
|
||||
|
||||
@property
|
||||
def st_bind(self) -> int:
|
||||
'''Returns STB_*.'''
|
||||
return self.st_info >> 4
|
||||
|
||||
class Verneed(ELFRecord):
|
||||
DEF = (BiStruct('HHIII'), ['vn_version', 'vn_cnt', 'vn_file', 'vn_aux', 'vn_next'])
|
||||
STRUCT = { ELFCLASS32: DEF, ELFCLASS64: DEF }
|
||||
|
||||
def __init__(self, data: bytes, offset: int, eh: ELFHeader) -> None:
|
||||
super().__init__(data, offset, eh, None)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'Verneed(vn_version={self.vn_version}, vn_cnt={self.vn_cnt}, vn_file={self.vn_file}, vn_aux={self.vn_aux}, vn_next={self.vn_next})'
|
||||
|
||||
class Vernaux(ELFRecord):
|
||||
DEF = (BiStruct('IHHII'), ['vna_hash', 'vna_flags', 'vna_other', 'vna_name', 'vna_next'])
|
||||
STRUCT = { ELFCLASS32: DEF, ELFCLASS64: DEF }
|
||||
|
||||
def __init__(self, data: bytes, offset: int, eh: ELFHeader, strings: bytes) -> None:
|
||||
super().__init__(data, offset, eh, None)
|
||||
self.name = _lookup_string(strings, self.vna_name)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'Veraux(vna_hash={self.vna_hash}, vna_flags={self.vna_flags}, vna_other={self.vna_other}, vna_name={self.vna_name}({self.name!r}), vna_next={self.vna_next})'
|
||||
|
||||
class DynTag(ELFRecord):
|
||||
STRUCT = {
|
||||
ELFCLASS32: (BiStruct('II'), ['d_tag', 'd_val']),
|
||||
ELFCLASS64: (BiStruct('QQ'), ['d_tag', 'd_val']),
|
||||
}
|
||||
|
||||
def __init__(self, data: bytes, offset: int, eh: ELFHeader, section: Section) -> None:
|
||||
super().__init__(data, offset, eh, section.sh_entsize)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'DynTag(d_tag={self.d_tag}, d_val={self.d_val})'
|
||||
|
||||
def _lookup_string(data: bytes, index: int) -> bytes:
|
||||
'''Look up string by offset in ELF string table.'''
|
||||
endx = data.find(b'\x00', index)
|
||||
assert endx != -1
|
||||
return data[index:endx]
|
||||
|
||||
VERSYM_S = BiStruct('H') # .gnu_version section has a single 16-bit integer per symbol in the linked section
|
||||
def _parse_symbol_table(section: Section, strings: bytes, eh: ELFHeader, versym: bytes, verneed: Dict[int, bytes]) -> List[Symbol]:
|
||||
'''Parse symbol table, return a list of symbols.'''
|
||||
data = section.contents()
|
||||
symbols = []
|
||||
versym_iter = (verneed.get(v[0]) for v in VERSYM_S[eh.ei_data].iter_unpack(versym))
|
||||
for ofs, version in zip(range(0, len(data), section.sh_entsize), versym_iter):
|
||||
symbols.append(Symbol(data, ofs, eh, section, strings, version))
|
||||
return symbols
|
||||
|
||||
def _parse_verneed(section: Section, strings: bytes, eh: ELFHeader) -> Dict[int, bytes]:
|
||||
'''Parse .gnu.version_r section, return a dictionary of {versym: 'GLIBC_...'}.'''
|
||||
data = section.contents()
|
||||
ofs = 0
|
||||
result = {}
|
||||
while True:
|
||||
verneed = Verneed(data, ofs, eh)
|
||||
aofs = ofs + verneed.vn_aux
|
||||
while True:
|
||||
vernaux = Vernaux(data, aofs, eh, strings)
|
||||
result[vernaux.vna_other] = vernaux.name
|
||||
if not vernaux.vna_next:
|
||||
break
|
||||
aofs += vernaux.vna_next
|
||||
|
||||
if not verneed.vn_next:
|
||||
break
|
||||
ofs += verneed.vn_next
|
||||
|
||||
return result
|
||||
|
||||
def _parse_dyn_tags(section: Section, strings: bytes, eh: ELFHeader) -> List[Tuple[int, Union[bytes, int]]]:
|
||||
'''Parse dynamic tags. Return array of tuples.'''
|
||||
data = section.contents()
|
||||
ofs = 0
|
||||
result = []
|
||||
for ofs in range(0, len(data), section.sh_entsize):
|
||||
tag = DynTag(data, ofs, eh, section)
|
||||
val = _lookup_string(strings, tag.d_val) if tag.d_tag in STRING_TAGS else tag.d_val
|
||||
result.append((tag.d_tag, val))
|
||||
|
||||
return result
|
||||
|
||||
class ELFFile:
|
||||
sections: List[Section]
|
||||
program_headers: List[ProgramHeader]
|
||||
dyn_symbols: List[Symbol]
|
||||
dyn_tags: List[Tuple[int, Union[bytes, int]]]
|
||||
|
||||
def __init__(self, data: bytes) -> None:
|
||||
self.data = data
|
||||
self.hdr = ELFHeader(self.data, 0)
|
||||
self._load_sections()
|
||||
self._load_program_headers()
|
||||
self._load_dyn_symbols()
|
||||
self._load_dyn_tags()
|
||||
self._section_to_segment_mapping()
|
||||
|
||||
def _load_sections(self) -> None:
|
||||
self.sections = []
|
||||
for idx in range(self.hdr.e_shnum):
|
||||
offset = self.hdr.e_shoff + idx * self.hdr.e_shentsize
|
||||
self.sections.append(Section(self.data, offset, self.hdr))
|
||||
|
||||
shstr = self.sections[self.hdr.e_shstrndx].contents()
|
||||
for section in self.sections:
|
||||
section.name = _lookup_string(shstr, section.sh_name)
|
||||
|
||||
def _load_program_headers(self) -> None:
|
||||
self.program_headers = []
|
||||
for idx in range(self.hdr.e_phnum):
|
||||
offset = self.hdr.e_phoff + idx * self.hdr.e_phentsize
|
||||
self.program_headers.append(ProgramHeader(self.data, offset, self.hdr))
|
||||
|
||||
def _load_dyn_symbols(self) -> None:
|
||||
# first, load 'verneed' section
|
||||
verneed = None
|
||||
for section in self.sections:
|
||||
if section.sh_type == SHT_GNU_verneed:
|
||||
strtab = self.sections[section.sh_link].contents() # associated string table
|
||||
assert verneed is None # only one section of this kind please
|
||||
verneed = _parse_verneed(section, strtab, self.hdr)
|
||||
assert verneed is not None
|
||||
|
||||
# then, correlate GNU versym sections with dynamic symbol sections
|
||||
versym = {}
|
||||
for section in self.sections:
|
||||
if section.sh_type == SHT_GNU_versym:
|
||||
versym[section.sh_link] = section
|
||||
|
||||
# finally, load dynsym sections
|
||||
self.dyn_symbols = []
|
||||
for idx, section in enumerate(self.sections):
|
||||
if section.sh_type == SHT_DYNSYM: # find dynamic symbol tables
|
||||
strtab_data = self.sections[section.sh_link].contents() # associated string table
|
||||
versym_data = versym[idx].contents() # associated symbol version table
|
||||
self.dyn_symbols += _parse_symbol_table(section, strtab_data, self.hdr, versym_data, verneed)
|
||||
|
||||
def _load_dyn_tags(self) -> None:
|
||||
self.dyn_tags = []
|
||||
for idx, section in enumerate(self.sections):
|
||||
if section.sh_type == SHT_DYNAMIC: # find dynamic tag tables
|
||||
strtab = self.sections[section.sh_link].contents() # associated string table
|
||||
self.dyn_tags += _parse_dyn_tags(section, strtab, self.hdr)
|
||||
|
||||
def _section_to_segment_mapping(self) -> None:
|
||||
for ph in self.program_headers:
|
||||
ph.sections = []
|
||||
for section in self.sections:
|
||||
if ph.p_vaddr <= section.sh_addr < (ph.p_vaddr + ph.p_memsz):
|
||||
ph.sections.append(section)
|
||||
|
||||
def query_dyn_tags(self, tag_in: int) -> List[Union[int, bytes]]:
|
||||
'''Return the values of all dyn tags with the specified tag.'''
|
||||
return [val for (tag, val) in self.dyn_tags if tag == tag_in]
|
||||
|
||||
|
||||
def load(filename: str) -> ELFFile:
|
||||
with open(filename, 'rb') as f:
|
||||
data = f.read()
|
||||
return ELFFile(data)
|
||||
261
contrib/devtools/security-check.py
Executable file
261
contrib/devtools/security-check.py
Executable file
|
|
@ -0,0 +1,261 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2015-2021 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Perform basic security checks on a series of executables.
|
||||
Exit status will be 0 if successful, and the program will be silent.
|
||||
Otherwise the exit status will be 1 and it will log which executables failed which checks.
|
||||
'''
|
||||
import sys
|
||||
from typing import List, Optional
|
||||
|
||||
import lief
|
||||
import pixie
|
||||
|
||||
def check_ELF_PIE(executable) -> bool:
|
||||
'''
|
||||
Check for position independent executable (PIE), allowing for address space randomization.
|
||||
'''
|
||||
elf = pixie.load(executable)
|
||||
return elf.hdr.e_type == pixie.ET_DYN
|
||||
|
||||
def check_ELF_NX(executable) -> bool:
|
||||
'''
|
||||
Check that no sections are writable and executable (including the stack)
|
||||
'''
|
||||
elf = pixie.load(executable)
|
||||
have_wx = False
|
||||
have_gnu_stack = False
|
||||
for ph in elf.program_headers:
|
||||
if ph.p_type == pixie.PT_GNU_STACK:
|
||||
have_gnu_stack = True
|
||||
if (ph.p_flags & pixie.PF_W) != 0 and (ph.p_flags & pixie.PF_X) != 0: # section is both writable and executable
|
||||
have_wx = True
|
||||
return have_gnu_stack and not have_wx
|
||||
|
||||
def check_ELF_RELRO(executable) -> bool:
|
||||
'''
|
||||
Check for read-only relocations.
|
||||
GNU_RELRO program header must exist
|
||||
Dynamic section must have BIND_NOW flag
|
||||
'''
|
||||
elf = pixie.load(executable)
|
||||
have_gnu_relro = False
|
||||
for ph in elf.program_headers:
|
||||
# Note: not checking p_flags == PF_R: here as linkers set the permission differently
|
||||
# This does not affect security: the permission flags of the GNU_RELRO program
|
||||
# header are ignored, the PT_LOAD header determines the effective permissions.
|
||||
# However, the dynamic linker need to write to this area so these are RW.
|
||||
# Glibc itself takes care of mprotecting this area R after relocations are finished.
|
||||
# See also https://marc.info/?l=binutils&m=1498883354122353
|
||||
if ph.p_type == pixie.PT_GNU_RELRO:
|
||||
have_gnu_relro = True
|
||||
|
||||
have_bindnow = False
|
||||
for flags in elf.query_dyn_tags(pixie.DT_FLAGS):
|
||||
assert isinstance(flags, int)
|
||||
if flags & pixie.DF_BIND_NOW:
|
||||
have_bindnow = True
|
||||
|
||||
return have_gnu_relro and have_bindnow
|
||||
|
||||
def check_ELF_Canary(executable) -> bool:
|
||||
'''
|
||||
Check for use of stack canary
|
||||
'''
|
||||
elf = pixie.load(executable)
|
||||
ok = False
|
||||
for symbol in elf.dyn_symbols:
|
||||
if symbol.name == b'__stack_chk_fail':
|
||||
ok = True
|
||||
return ok
|
||||
|
||||
def check_ELF_separate_code(executable):
|
||||
'''
|
||||
Check that sections are appropriately separated in virtual memory,
|
||||
based on their permissions. This checks for missing -Wl,-z,separate-code
|
||||
and potentially other problems.
|
||||
'''
|
||||
elf = pixie.load(executable)
|
||||
R = pixie.PF_R
|
||||
W = pixie.PF_W
|
||||
E = pixie.PF_X
|
||||
EXPECTED_FLAGS = {
|
||||
# Read + execute
|
||||
b'.init': R | E,
|
||||
b'.plt': R | E,
|
||||
b'.plt.got': R | E,
|
||||
b'.plt.sec': R | E,
|
||||
b'.text': R | E,
|
||||
b'.fini': R | E,
|
||||
# Read-only data
|
||||
b'.interp': R,
|
||||
b'.note.gnu.property': R,
|
||||
b'.note.gnu.build-id': R,
|
||||
b'.note.ABI-tag': R,
|
||||
b'.gnu.hash': R,
|
||||
b'.dynsym': R,
|
||||
b'.dynstr': R,
|
||||
b'.gnu.version': R,
|
||||
b'.gnu.version_r': R,
|
||||
b'.rela.dyn': R,
|
||||
b'.rela.plt': R,
|
||||
b'.rodata': R,
|
||||
b'.eh_frame_hdr': R,
|
||||
b'.eh_frame': R,
|
||||
b'.qtmetadata': R,
|
||||
b'.gcc_except_table': R,
|
||||
b'.stapsdt.base': R,
|
||||
# Writable data
|
||||
b'.init_array': R | W,
|
||||
b'.fini_array': R | W,
|
||||
b'.dynamic': R | W,
|
||||
b'.got': R | W,
|
||||
b'.data': R | W,
|
||||
b'.bss': R | W,
|
||||
}
|
||||
if elf.hdr.e_machine == pixie.EM_PPC64:
|
||||
# .plt is RW on ppc64 even with separate-code
|
||||
EXPECTED_FLAGS[b'.plt'] = R | W
|
||||
# For all LOAD program headers get mapping to the list of sections,
|
||||
# and for each section, remember the flags of the associated program header.
|
||||
flags_per_section = {}
|
||||
for ph in elf.program_headers:
|
||||
if ph.p_type == pixie.PT_LOAD:
|
||||
for section in ph.sections:
|
||||
assert(section.name not in flags_per_section)
|
||||
flags_per_section[section.name] = ph.p_flags
|
||||
# Spot-check ELF LOAD program header flags per section
|
||||
# If these sections exist, check them against the expected R/W/E flags
|
||||
for (section, flags) in flags_per_section.items():
|
||||
if section in EXPECTED_FLAGS:
|
||||
if EXPECTED_FLAGS[section] != flags:
|
||||
return False
|
||||
return True
|
||||
|
||||
def check_PE_DYNAMIC_BASE(executable) -> bool:
|
||||
'''PIE: DllCharacteristics bit 0x40 signifies dynamicbase (ASLR)'''
|
||||
binary = lief.parse(executable)
|
||||
return lief.PE.DLL_CHARACTERISTICS.DYNAMIC_BASE in binary.optional_header.dll_characteristics_lists
|
||||
|
||||
# Must support high-entropy 64-bit address space layout randomization
|
||||
# in addition to DYNAMIC_BASE to have secure ASLR.
|
||||
def check_PE_HIGH_ENTROPY_VA(executable) -> bool:
|
||||
'''PIE: DllCharacteristics bit 0x20 signifies high-entropy ASLR'''
|
||||
binary = lief.parse(executable)
|
||||
return lief.PE.DLL_CHARACTERISTICS.HIGH_ENTROPY_VA in binary.optional_header.dll_characteristics_lists
|
||||
|
||||
def check_PE_RELOC_SECTION(executable) -> bool:
|
||||
'''Check for a reloc section. This is required for functional ASLR.'''
|
||||
binary = lief.parse(executable)
|
||||
return binary.has_relocations
|
||||
|
||||
def check_MACHO_NOUNDEFS(executable) -> bool:
|
||||
'''
|
||||
Check for no undefined references.
|
||||
'''
|
||||
binary = lief.parse(executable)
|
||||
return binary.header.has(lief.MachO.HEADER_FLAGS.NOUNDEFS)
|
||||
|
||||
def check_MACHO_LAZY_BINDINGS(executable) -> bool:
|
||||
'''
|
||||
Check for no lazy bindings.
|
||||
We don't use or check for MH_BINDATLOAD. See #18295.
|
||||
'''
|
||||
binary = lief.parse(executable)
|
||||
return binary.dyld_info.lazy_bind == (0,0)
|
||||
|
||||
def check_MACHO_Canary(executable) -> bool:
|
||||
'''
|
||||
Check for use of stack canary
|
||||
'''
|
||||
binary = lief.parse(executable)
|
||||
return binary.has_symbol('___stack_chk_fail')
|
||||
|
||||
def check_PIE(executable) -> bool:
|
||||
'''
|
||||
Check for position independent executable (PIE),
|
||||
allowing for address space randomization.
|
||||
'''
|
||||
binary = lief.parse(executable)
|
||||
return binary.is_pie
|
||||
|
||||
def check_NX(executable) -> bool:
|
||||
'''
|
||||
Check for no stack execution
|
||||
'''
|
||||
binary = lief.parse(executable)
|
||||
return binary.has_nx
|
||||
|
||||
def check_control_flow(executable) -> bool:
|
||||
'''
|
||||
Check for control flow instrumentation
|
||||
'''
|
||||
binary = lief.parse(executable)
|
||||
|
||||
content = binary.get_content_from_virtual_address(binary.entrypoint, 4, lief.Binary.VA_TYPES.AUTO)
|
||||
|
||||
if content == [243, 15, 30, 250]: # endbr64
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
CHECKS = {
|
||||
'ELF': [
|
||||
('PIE', check_ELF_PIE),
|
||||
('NX', check_ELF_NX),
|
||||
('RELRO', check_ELF_RELRO),
|
||||
('Canary', check_ELF_Canary),
|
||||
('separate_code', check_ELF_separate_code),
|
||||
],
|
||||
'PE': [
|
||||
('PIE', check_PIE),
|
||||
('DYNAMIC_BASE', check_PE_DYNAMIC_BASE),
|
||||
('HIGH_ENTROPY_VA', check_PE_HIGH_ENTROPY_VA),
|
||||
('NX', check_NX),
|
||||
('RELOC_SECTION', check_PE_RELOC_SECTION)
|
||||
],
|
||||
'MACHO': [
|
||||
('PIE', check_PIE),
|
||||
('NOUNDEFS', check_MACHO_NOUNDEFS),
|
||||
('NX', check_NX),
|
||||
('LAZY_BINDINGS', check_MACHO_LAZY_BINDINGS),
|
||||
('Canary', check_MACHO_Canary),
|
||||
('CONTROL_FLOW', check_control_flow),
|
||||
]
|
||||
}
|
||||
|
||||
def identify_executable(executable) -> Optional[str]:
|
||||
with open(filename, 'rb') as f:
|
||||
magic = f.read(4)
|
||||
if magic.startswith(b'MZ'):
|
||||
return 'PE'
|
||||
elif magic.startswith(b'\x7fELF'):
|
||||
return 'ELF'
|
||||
elif magic.startswith(b'\xcf\xfa'):
|
||||
return 'MACHO'
|
||||
return None
|
||||
|
||||
if __name__ == '__main__':
|
||||
retval: int = 0
|
||||
for filename in sys.argv[1:]:
|
||||
try:
|
||||
etype = identify_executable(filename)
|
||||
if etype is None:
|
||||
print(f'{filename}: unknown format')
|
||||
retval = 1
|
||||
continue
|
||||
|
||||
failed: List[str] = []
|
||||
for (name, func) in CHECKS[etype]:
|
||||
if not func(filename):
|
||||
failed.append(name)
|
||||
if failed:
|
||||
print(f'{filename}: failed {" ".join(failed)}')
|
||||
retval = 1
|
||||
except IOError:
|
||||
print(f'{filename}: cannot open')
|
||||
retval = 1
|
||||
sys.exit(retval)
|
||||
|
||||
10
contrib/devtools/split-debug.sh.in
Normal file
10
contrib/devtools/split-debug.sh.in
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
#!/bin/sh
|
||||
set -e
|
||||
if [ $# -ne 3 ];
|
||||
then echo "usage: $0 <input> <stripped-binary> <debug-binary>"
|
||||
fi
|
||||
|
||||
@OBJCOPY@ --enable-deterministic-archives -p --only-keep-debug $1 $3
|
||||
@OBJCOPY@ --enable-deterministic-archives -p --strip-debug $1 $2
|
||||
@STRIP@ --enable-deterministic-archives -p -s $2
|
||||
@OBJCOPY@ --enable-deterministic-archives -p --add-gnu-debuglink=$3 $2
|
||||
308
contrib/devtools/symbol-check.py
Executable file
308
contrib/devtools/symbol-check.py
Executable file
|
|
@ -0,0 +1,308 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2014 Wladimir J. van der Laan
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
A script to check that release executables only contain certain symbols
|
||||
and are only linked against allowed libraries.
|
||||
|
||||
Example usage:
|
||||
|
||||
find ../path/to/binaries -type f -executable | xargs python3 contrib/devtools/symbol-check.py
|
||||
'''
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import List, Optional
|
||||
|
||||
import lief
|
||||
import pixie
|
||||
|
||||
from utils import determine_wellknown_cmd
|
||||
|
||||
# Debian 8 (Jessie) EOL: 2020. https://wiki.debian.org/DebianReleases#Production_Releases
|
||||
#
|
||||
# - g++ version 4.9.2 (https://packages.debian.org/search?suite=jessie&arch=any&searchon=names&keywords=g%2B%2B)
|
||||
# - libc version 2.19 (https://packages.debian.org/search?suite=jessie&arch=any&searchon=names&keywords=libc6)
|
||||
#
|
||||
# Ubuntu 16.04 (Xenial) EOL: 2024. https://wiki.ubuntu.com/Releases
|
||||
#
|
||||
# - g++ version 5.3.1 (https://packages.ubuntu.com/search?keywords=g%2B%2B&searchon=names&suite=xenial§ion=all)
|
||||
# - libc version 2.23.0 (https://packages.ubuntu.com/search?keywords=libc6&searchon=names&suite=xenial§ion=all)
|
||||
#
|
||||
# CentOS 7 EOL: 2024. https://wiki.centos.org/FAQ/General
|
||||
#
|
||||
# - g++ version 4.8.5 (http://mirror.centos.org/centos/7/os/x86_64/Packages/)
|
||||
# - libc version 2.17 (http://mirror.centos.org/centos/7/os/x86_64/Packages/)
|
||||
#
|
||||
# Taking the minimum of these as our target.
|
||||
#
|
||||
# According to GNU ABI document (https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html) this corresponds to:
|
||||
# GCC 4.8.5: GCC_4.8.0
|
||||
# (glibc) GLIBC_2_17
|
||||
#
|
||||
MAX_VERSIONS = {
|
||||
'GCC': (4,8,0),
|
||||
'GLIBC': {
|
||||
pixie.EM_386: (2,18),
|
||||
pixie.EM_X86_64: (2,18),
|
||||
pixie.EM_ARM: (2,18),
|
||||
pixie.EM_AARCH64:(2,18),
|
||||
pixie.EM_PPC64: (2,18),
|
||||
pixie.EM_RISCV: (2,27),
|
||||
},
|
||||
'LIBATOMIC': (1,0),
|
||||
'V': (0,5,0), # xkb (digibyte-qt only)
|
||||
}
|
||||
# See here for a description of _IO_stdin_used:
|
||||
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261#109
|
||||
|
||||
# Ignore symbols that are exported as part of every executable
|
||||
IGNORE_EXPORTS = {
|
||||
'_edata', '_end', '__end__', '_init', '__bss_start', '__bss_start__', '_bss_end__', '__bss_end__', '_fini', '_IO_stdin_used', 'stdin', 'stdout', 'stderr',
|
||||
'environ', '_environ', '__environ',
|
||||
}
|
||||
|
||||
# Allowed NEEDED libraries
|
||||
ELF_ALLOWED_LIBRARIES = {
|
||||
# digibyted and digibyte-qt
|
||||
'libgcc_s.so.1', # GCC base support
|
||||
'libc.so.6', # C library
|
||||
'libpthread.so.0', # threading
|
||||
'libm.so.6', # math library
|
||||
'librt.so.1', # real-time (clock)
|
||||
'libatomic.so.1',
|
||||
'ld-linux-x86-64.so.2', # 64-bit dynamic linker
|
||||
'ld-linux.so.2', # 32-bit dynamic linker
|
||||
'ld-linux-aarch64.so.1', # 64-bit ARM dynamic linker
|
||||
'ld-linux-armhf.so.3', # 32-bit ARM dynamic linker
|
||||
'ld64.so.1', # POWER64 ABIv1 dynamic linker
|
||||
'ld64.so.2', # POWER64 ABIv2 dynamic linker
|
||||
'ld-linux-riscv64-lp64d.so.1', # 64-bit RISC-V dynamic linker
|
||||
# digibyte-qt only
|
||||
'libxcb.so.1', # part of X11
|
||||
'libxkbcommon.so.0', # keyboard keymapping
|
||||
'libxkbcommon-x11.so.0', # keyboard keymapping
|
||||
'libfontconfig.so.1', # font support
|
||||
'libfreetype.so.6', # font parsing
|
||||
'libdl.so.2', # programming interface to dynamic linker
|
||||
'libxcb-icccm.so.4',
|
||||
'libxcb-image.so.0',
|
||||
'libxcb-shm.so.0',
|
||||
'libxcb-keysyms.so.1',
|
||||
'libxcb-randr.so.0',
|
||||
'libxcb-render-util.so.0',
|
||||
'libxcb-render.so.0',
|
||||
'libxcb-shape.so.0',
|
||||
'libxcb-sync.so.1',
|
||||
'libxcb-xfixes.so.0',
|
||||
'libxcb-xinerama.so.0',
|
||||
'libxcb-xkb.so.1',
|
||||
}
|
||||
|
||||
MACHO_ALLOWED_LIBRARIES = {
|
||||
# digibyted and digibyte-qt
|
||||
'libc++.1.dylib', # C++ Standard Library
|
||||
'libSystem.B.dylib', # libc, libm, libpthread, libinfo
|
||||
# digibyte-qt only
|
||||
'AppKit', # user interface
|
||||
'ApplicationServices', # common application tasks.
|
||||
'Carbon', # deprecated c back-compat API
|
||||
'ColorSync',
|
||||
'CoreFoundation', # low level func, data types
|
||||
'CoreGraphics', # 2D rendering
|
||||
'CoreServices', # operating system services
|
||||
'CoreText', # interface for laying out text and handling fonts.
|
||||
'CoreVideo', # video processing
|
||||
'Foundation', # base layer functionality for apps/frameworks
|
||||
'ImageIO', # read and write image file formats.
|
||||
'IOKit', # user-space access to hardware devices and drivers.
|
||||
'IOSurface', # cross process image/drawing buffers
|
||||
'libobjc.A.dylib', # Objective-C runtime library
|
||||
'Metal', # 3D graphics
|
||||
'Security', # access control and authentication
|
||||
'QuartzCore', # animation
|
||||
}
|
||||
|
||||
PE_ALLOWED_LIBRARIES = {
|
||||
'ADVAPI32.dll', # security & registry
|
||||
'IPHLPAPI.DLL', # IP helper API
|
||||
'KERNEL32.dll', # win32 base APIs
|
||||
'msvcrt.dll', # C standard library for MSVC
|
||||
'SHELL32.dll', # shell API
|
||||
'USER32.dll', # user interface
|
||||
'WS2_32.dll', # sockets
|
||||
# digibyte-qt only
|
||||
'dwmapi.dll', # desktop window manager
|
||||
'GDI32.dll', # graphics device interface
|
||||
'IMM32.dll', # input method editor
|
||||
'NETAPI32.dll',
|
||||
'ole32.dll', # component object model
|
||||
'OLEAUT32.dll', # OLE Automation API
|
||||
'SHLWAPI.dll', # light weight shell API
|
||||
'USERENV.dll',
|
||||
'UxTheme.dll',
|
||||
'VERSION.dll', # version checking
|
||||
'WINMM.dll', # WinMM audio API
|
||||
'WTSAPI32.dll',
|
||||
}
|
||||
|
||||
class CPPFilt(object):
|
||||
'''
|
||||
Demangle C++ symbol names.
|
||||
|
||||
Use a pipe to the 'c++filt' command.
|
||||
'''
|
||||
def __init__(self):
|
||||
self.proc = subprocess.Popen(determine_wellknown_cmd('CPPFILT', 'c++filt'), stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True)
|
||||
|
||||
def __call__(self, mangled):
|
||||
self.proc.stdin.write(mangled + '\n')
|
||||
self.proc.stdin.flush()
|
||||
return self.proc.stdout.readline().rstrip()
|
||||
|
||||
def close(self):
|
||||
self.proc.stdin.close()
|
||||
self.proc.stdout.close()
|
||||
self.proc.wait()
|
||||
|
||||
def check_version(max_versions, version, arch) -> bool:
|
||||
if '_' in version:
|
||||
(lib, _, ver) = version.rpartition('_')
|
||||
else:
|
||||
lib = version
|
||||
ver = '0'
|
||||
ver = tuple([int(x) for x in ver.split('.')])
|
||||
if not lib in max_versions:
|
||||
return False
|
||||
if isinstance(max_versions[lib], tuple):
|
||||
return ver <= max_versions[lib]
|
||||
else:
|
||||
return ver <= max_versions[lib][arch]
|
||||
|
||||
def check_imported_symbols(filename) -> bool:
|
||||
elf = pixie.load(filename)
|
||||
cppfilt = CPPFilt()
|
||||
ok: bool = True
|
||||
|
||||
for symbol in elf.dyn_symbols:
|
||||
if not symbol.is_import:
|
||||
continue
|
||||
sym = symbol.name.decode()
|
||||
version = symbol.version.decode() if symbol.version is not None else None
|
||||
if version and not check_version(MAX_VERSIONS, version, elf.hdr.e_machine):
|
||||
print('{}: symbol {} from unsupported version {}'.format(filename, cppfilt(sym), version))
|
||||
ok = False
|
||||
return ok
|
||||
|
||||
def check_exported_symbols(filename) -> bool:
|
||||
elf = pixie.load(filename)
|
||||
cppfilt = CPPFilt()
|
||||
ok: bool = True
|
||||
for symbol in elf.dyn_symbols:
|
||||
if not symbol.is_export:
|
||||
continue
|
||||
sym = symbol.name.decode()
|
||||
if elf.hdr.e_machine == pixie.EM_RISCV or sym in IGNORE_EXPORTS:
|
||||
continue
|
||||
print('{}: export of symbol {} not allowed'.format(filename, cppfilt(sym)))
|
||||
ok = False
|
||||
return ok
|
||||
|
||||
def check_ELF_libraries(filename) -> bool:
|
||||
ok: bool = True
|
||||
elf = pixie.load(filename)
|
||||
for library_name in elf.query_dyn_tags(pixie.DT_NEEDED):
|
||||
assert(isinstance(library_name, bytes))
|
||||
if library_name.decode() not in ELF_ALLOWED_LIBRARIES:
|
||||
print('{}: NEEDED library {} is not allowed'.format(filename, library_name.decode()))
|
||||
ok = False
|
||||
return ok
|
||||
|
||||
def check_MACHO_libraries(filename) -> bool:
|
||||
ok: bool = True
|
||||
binary = lief.parse(filename)
|
||||
for dylib in binary.libraries:
|
||||
split = dylib.name.split('/')
|
||||
if split[-1] not in MACHO_ALLOWED_LIBRARIES:
|
||||
print(f'{split[-1]} is not in ALLOWED_LIBRARIES!')
|
||||
ok = False
|
||||
return ok
|
||||
|
||||
def check_MACHO_min_os(filename) -> bool:
|
||||
binary = lief.parse(filename)
|
||||
if binary.build_version.minos == [10,15,0]:
|
||||
return True
|
||||
return False
|
||||
|
||||
def check_MACHO_sdk(filename) -> bool:
|
||||
binary = lief.parse(filename)
|
||||
if binary.build_version.sdk == [11, 0, 0]:
|
||||
return True
|
||||
return False
|
||||
|
||||
def check_PE_libraries(filename) -> bool:
|
||||
ok: bool = True
|
||||
binary = lief.parse(filename)
|
||||
for dylib in binary.libraries:
|
||||
if dylib not in PE_ALLOWED_LIBRARIES:
|
||||
print(f'{dylib} is not in ALLOWED_LIBRARIES!')
|
||||
ok = False
|
||||
return ok
|
||||
|
||||
def check_PE_subsystem_version(filename) -> bool:
|
||||
binary = lief.parse(filename)
|
||||
major: int = binary.optional_header.major_subsystem_version
|
||||
minor: int = binary.optional_header.minor_subsystem_version
|
||||
if major == 6 and minor == 1:
|
||||
return True
|
||||
return False
|
||||
|
||||
CHECKS = {
|
||||
'ELF': [
|
||||
('IMPORTED_SYMBOLS', check_imported_symbols),
|
||||
('EXPORTED_SYMBOLS', check_exported_symbols),
|
||||
('LIBRARY_DEPENDENCIES', check_ELF_libraries)
|
||||
],
|
||||
'MACHO': [
|
||||
('DYNAMIC_LIBRARIES', check_MACHO_libraries),
|
||||
('MIN_OS', check_MACHO_min_os),
|
||||
('SDK', check_MACHO_sdk),
|
||||
],
|
||||
'PE' : [
|
||||
('DYNAMIC_LIBRARIES', check_PE_libraries),
|
||||
('SUBSYSTEM_VERSION', check_PE_subsystem_version),
|
||||
]
|
||||
}
|
||||
|
||||
def identify_executable(executable) -> Optional[str]:
|
||||
with open(filename, 'rb') as f:
|
||||
magic = f.read(4)
|
||||
if magic.startswith(b'MZ'):
|
||||
return 'PE'
|
||||
elif magic.startswith(b'\x7fELF'):
|
||||
return 'ELF'
|
||||
elif magic.startswith(b'\xcf\xfa'):
|
||||
return 'MACHO'
|
||||
return None
|
||||
|
||||
if __name__ == '__main__':
|
||||
retval: int = 0
|
||||
for filename in sys.argv[1:]:
|
||||
try:
|
||||
etype = identify_executable(filename)
|
||||
if etype is None:
|
||||
print(f'{filename}: unknown format')
|
||||
retval = 1
|
||||
continue
|
||||
|
||||
failed: List[str] = []
|
||||
for (name, func) in CHECKS[etype]:
|
||||
if not func(filename):
|
||||
failed.append(name)
|
||||
if failed:
|
||||
print(f'{filename}: failed {" ".join(failed)}')
|
||||
retval = 1
|
||||
except IOError:
|
||||
print(f'{filename}: cannot open')
|
||||
retval = 1
|
||||
sys.exit(retval)
|
||||
112
contrib/devtools/test-security-check.py
Executable file
112
contrib/devtools/test-security-check.py
Executable file
|
|
@ -0,0 +1,112 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2015-2021 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Test script for security-check.py
|
||||
'''
|
||||
import lief #type:ignore
|
||||
import os
|
||||
import subprocess
|
||||
from typing import List
|
||||
import unittest
|
||||
|
||||
from utils import determine_wellknown_cmd
|
||||
|
||||
def write_testcode(filename):
|
||||
with open(filename, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#include <stdio.h>
|
||||
int main()
|
||||
{
|
||||
printf("the quick brown fox jumps over the lazy god\\n");
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
def clean_files(source, executable):
|
||||
os.remove(source)
|
||||
os.remove(executable)
|
||||
|
||||
def call_security_check(cc, source, executable, options):
|
||||
subprocess.run([*cc,source,'-o',executable] + options, check=True)
|
||||
p = subprocess.run(['./contrib/devtools/security-check.py',executable], stdout=subprocess.PIPE, universal_newlines=True)
|
||||
return (p.returncode, p.stdout.rstrip())
|
||||
|
||||
def get_arch(cc, source, executable):
|
||||
subprocess.run([*cc, source, '-o', executable], check=True)
|
||||
binary = lief.parse(executable)
|
||||
arch = binary.abstract.header.architecture
|
||||
os.remove(executable)
|
||||
return arch
|
||||
|
||||
class TestSecurityChecks(unittest.TestCase):
|
||||
def test_ELF(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1'
|
||||
cc = determine_wellknown_cmd('CC', 'gcc')
|
||||
write_testcode(source)
|
||||
arch = get_arch(cc, source, executable)
|
||||
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-zexecstack','-fno-stack-protector','-Wl,-znorelro','-no-pie','-fno-PIE', '-Wl,-z,separate-code']),
|
||||
(1, executable+': failed PIE NX RELRO Canary'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fno-stack-protector','-Wl,-znorelro','-no-pie','-fno-PIE', '-Wl,-z,separate-code']),
|
||||
(1, executable+': failed PIE RELRO Canary'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-znorelro','-no-pie','-fno-PIE', '-Wl,-z,separate-code']),
|
||||
(1, executable+': failed PIE RELRO'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-znorelro','-pie','-fPIE', '-Wl,-z,separate-code']),
|
||||
(1, executable+': failed RELRO'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-zrelro','-Wl,-z,now','-pie','-fPIE', '-Wl,-z,noseparate-code']),
|
||||
(1, executable+': failed separate_code'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-zrelro','-Wl,-z,now','-pie','-fPIE', '-Wl,-z,separate-code']),
|
||||
(0, ''))
|
||||
|
||||
clean_files(source, executable)
|
||||
|
||||
def test_PE(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1.exe'
|
||||
cc = determine_wellknown_cmd('CC', 'x86_64-w64-mingw32-gcc')
|
||||
write_testcode(source)
|
||||
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--no-nxcompat','-Wl,--disable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-no-pie','-fno-PIE']),
|
||||
(1, executable+': failed PIE DYNAMIC_BASE HIGH_ENTROPY_VA NX RELOC_SECTION'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--disable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-no-pie','-fno-PIE']),
|
||||
(1, executable+': failed PIE DYNAMIC_BASE HIGH_ENTROPY_VA RELOC_SECTION'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--enable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-no-pie','-fno-PIE']),
|
||||
(1, executable+': failed PIE DYNAMIC_BASE HIGH_ENTROPY_VA'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--enable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-pie','-fPIE']),
|
||||
(1, executable+': failed PIE DYNAMIC_BASE HIGH_ENTROPY_VA')) # -pie -fPIE does nothing unless --dynamicbase is also supplied
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--enable-reloc-section','-Wl,--dynamicbase','-Wl,--no-high-entropy-va','-pie','-fPIE']),
|
||||
(1, executable+': failed HIGH_ENTROPY_VA'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--enable-reloc-section','-Wl,--dynamicbase','-Wl,--high-entropy-va','-pie','-fPIE']),
|
||||
(0, ''))
|
||||
|
||||
clean_files(source, executable)
|
||||
|
||||
def test_MACHO(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1'
|
||||
cc = determine_wellknown_cmd('CC', 'clang')
|
||||
write_testcode(source)
|
||||
arch = get_arch(cc, source, executable)
|
||||
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector']),
|
||||
(1, executable+': failed PIE NOUNDEFS NX LAZY_BINDINGS Canary CONTROL_FLOW'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fstack-protector-all']),
|
||||
(1, executable+': failed PIE NOUNDEFS NX LAZY_BINDINGS CONTROL_FLOW'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-fstack-protector-all']),
|
||||
(1, executable+': failed PIE NOUNDEFS LAZY_BINDINGS CONTROL_FLOW'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all']),
|
||||
(1, executable+': failed PIE LAZY_BINDINGS CONTROL_FLOW'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all']),
|
||||
(1, executable+': failed PIE CONTROL_FLOW'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
|
||||
(1, executable+': failed PIE'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
|
||||
(0, ''))
|
||||
|
||||
clean_files(source, executable)
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
204
contrib/devtools/test-symbol-check.py
Executable file
204
contrib/devtools/test-symbol-check.py
Executable file
|
|
@ -0,0 +1,204 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2020-2021 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Test script for symbol-check.py
|
||||
'''
|
||||
import os
|
||||
import subprocess
|
||||
from typing import List
|
||||
import unittest
|
||||
|
||||
from utils import determine_wellknown_cmd
|
||||
|
||||
def call_symbol_check(cc: List[str], source, executable, options):
|
||||
# This should behave the same as AC_TRY_LINK, so arrange well-known flags
|
||||
# in the same order as autoconf would.
|
||||
#
|
||||
# See the definitions for ac_link in autoconf's lib/autoconf/c.m4 file for
|
||||
# reference.
|
||||
env_flags: List[str] = []
|
||||
for var in ['CFLAGS', 'CPPFLAGS', 'LDFLAGS']:
|
||||
env_flags += filter(None, os.environ.get(var, '').split(' '))
|
||||
|
||||
subprocess.run([*cc,source,'-o',executable] + env_flags + options, check=True)
|
||||
p = subprocess.run(['./contrib/devtools/symbol-check.py',executable], stdout=subprocess.PIPE, universal_newlines=True)
|
||||
os.remove(source)
|
||||
os.remove(executable)
|
||||
return (p.returncode, p.stdout.rstrip())
|
||||
|
||||
def get_machine(cc: List[str]):
|
||||
p = subprocess.run([*cc,'-dumpmachine'], stdout=subprocess.PIPE, universal_newlines=True)
|
||||
return p.stdout.rstrip()
|
||||
|
||||
class TestSymbolChecks(unittest.TestCase):
|
||||
def test_ELF(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1'
|
||||
cc = determine_wellknown_cmd('CC', 'gcc')
|
||||
|
||||
# there's no way to do this test for RISC-V at the moment; we build for
|
||||
# RISC-V in a glibc 2.27 environment and we allow all symbols from 2.27.
|
||||
if 'riscv' in get_machine(cc):
|
||||
self.skipTest("test not available for RISC-V")
|
||||
|
||||
# nextup was introduced in GLIBC 2.24, so is newer than our supported
|
||||
# glibc (2.18), and available in our release build environment (2.24).
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#define _GNU_SOURCE
|
||||
#include <math.h>
|
||||
|
||||
double nextup(double x);
|
||||
|
||||
int main()
|
||||
{
|
||||
nextup(3.14);
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, ['-lm']),
|
||||
(1, executable + ': symbol nextup from unsupported version GLIBC_2.24\n' +
|
||||
executable + ': failed IMPORTED_SYMBOLS'))
|
||||
|
||||
# -lutil is part of the libc6 package so a safe bet that it's installed
|
||||
# it's also out of context enough that it's unlikely to ever become a real dependency
|
||||
source = 'test2.c'
|
||||
executable = 'test2'
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#include <utmp.h>
|
||||
|
||||
int main()
|
||||
{
|
||||
login(0);
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, ['-lutil']),
|
||||
(1, executable + ': NEEDED library libutil.so.1 is not allowed\n' +
|
||||
executable + ': failed LIBRARY_DEPENDENCIES'))
|
||||
|
||||
# finally, check a simple conforming binary
|
||||
source = 'test3.c'
|
||||
executable = 'test3'
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#include <stdio.h>
|
||||
|
||||
int main()
|
||||
{
|
||||
printf("42");
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, []),
|
||||
(0, ''))
|
||||
|
||||
def test_MACHO(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1'
|
||||
cc = determine_wellknown_cmd('CC', 'clang')
|
||||
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#include <expat.h>
|
||||
|
||||
int main()
|
||||
{
|
||||
XML_ExpatVersion();
|
||||
return 0;
|
||||
}
|
||||
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, ['-lexpat', '-Wl,-platform_version','-Wl,macos', '-Wl,11.4', '-Wl,11.4']),
|
||||
(1, 'libexpat.1.dylib is not in ALLOWED_LIBRARIES!\n' +
|
||||
f'{executable}: failed DYNAMIC_LIBRARIES MIN_OS SDK'))
|
||||
|
||||
source = 'test2.c'
|
||||
executable = 'test2'
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#include <CoreGraphics/CoreGraphics.h>
|
||||
|
||||
int main()
|
||||
{
|
||||
CGMainDisplayID();
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, ['-framework', 'CoreGraphics', '-Wl,-platform_version','-Wl,macos', '-Wl,11.4', '-Wl,11.4']),
|
||||
(1, f'{executable}: failed MIN_OS SDK'))
|
||||
|
||||
source = 'test3.c'
|
||||
executable = 'test3'
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
int main()
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, ['-Wl,-platform_version','-Wl,macos', '-Wl,10.15', '-Wl,11.4']),
|
||||
(1, f'{executable}: failed SDK'))
|
||||
|
||||
def test_PE(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1.exe'
|
||||
cc = determine_wellknown_cmd('CC', 'x86_64-w64-mingw32-gcc')
|
||||
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#include <pdh.h>
|
||||
|
||||
int main()
|
||||
{
|
||||
PdhConnectMachineA(NULL);
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, ['-lpdh', '-Wl,--major-subsystem-version', '-Wl,6', '-Wl,--minor-subsystem-version', '-Wl,1']),
|
||||
(1, 'pdh.dll is not in ALLOWED_LIBRARIES!\n' +
|
||||
executable + ': failed DYNAMIC_LIBRARIES'))
|
||||
|
||||
source = 'test2.c'
|
||||
executable = 'test2.exe'
|
||||
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
int main()
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, ['-Wl,--major-subsystem-version', '-Wl,9', '-Wl,--minor-subsystem-version', '-Wl,9']),
|
||||
(1, executable + ': failed SUBSYSTEM_VERSION'))
|
||||
|
||||
source = 'test3.c'
|
||||
executable = 'test3.exe'
|
||||
with open(source, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#include <combaseapi.h>
|
||||
|
||||
int main()
|
||||
{
|
||||
CoFreeUnusedLibrariesEx(0,0);
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
self.assertEqual(call_symbol_check(cc, source, executable, ['-lole32', '-Wl,--major-subsystem-version', '-Wl,6', '-Wl,--minor-subsystem-version', '-Wl,1']),
|
||||
(0, ''))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
151
contrib/devtools/test_deterministic_coverage.sh
Executable file
151
contrib/devtools/test_deterministic_coverage.sh
Executable file
|
|
@ -0,0 +1,151 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# Copyright (c) 2019-2020 The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
#
|
||||
# Test for deterministic coverage across unit test runs.
|
||||
|
||||
export LC_ALL=C
|
||||
|
||||
# Use GCOV_EXECUTABLE="gcov" if compiling with gcc.
|
||||
# Use GCOV_EXECUTABLE="llvm-cov gcov" if compiling with clang.
|
||||
GCOV_EXECUTABLE="gcov"
|
||||
|
||||
# Disable tests known to cause non-deterministic behaviour and document the source or point of non-determinism.
|
||||
NON_DETERMINISTIC_TESTS=(
|
||||
"blockfilter_index_tests/blockfilter_index_initial_sync" # src/checkqueue.h: In CCheckQueue::Loop(): while (queue.empty()) { ... }
|
||||
"coinselector_tests/knapsack_solver_test" # coinselector_tests.cpp: if (equal_sets(setCoinsRet, setCoinsRet2))
|
||||
"fs_tests/fsbridge_fstream" # deterministic test failure?
|
||||
"miner_tests/CreateNewBlock_validity" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"scheduler_tests/manythreads" # scheduler.cpp: CScheduler::serviceQueue()
|
||||
"scheduler_tests/singlethreadedscheduler_ordered" # scheduler.cpp: CScheduler::serviceQueue()
|
||||
"txvalidationcache_tests/checkinputs_test" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"txvalidationcache_tests/tx_mempool_block_doublespend" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"txindex_tests/txindex_initial_sync" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"txvalidation_tests/tx_mempool_reject_coinbase" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"validation_block_tests/processnewblock_signals_ordering" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"wallet_tests/coin_mark_dirty_immature_credit" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"wallet_tests/dummy_input_size_test" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"wallet_tests/importmulti_rescan" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"wallet_tests/importwallet_rescan" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"wallet_tests/ListCoins" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"wallet_tests/scan_for_wallet_transactions" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
"wallet_tests/wallet_disableprivkeys" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
|
||||
)
|
||||
|
||||
TEST_DIGIBYTE_BINARY="src/test/test_digibyte"
|
||||
|
||||
print_usage() {
|
||||
echo "Usage: $0 [custom test filter (default: all but known non-deterministic tests)] [number of test runs (default: 2)]"
|
||||
}
|
||||
|
||||
N_TEST_RUNS=2
|
||||
BOOST_TEST_RUN_FILTERS=""
|
||||
if [[ $# != 0 ]]; then
|
||||
if [[ $1 == "--help" ]]; then
|
||||
print_usage
|
||||
exit
|
||||
fi
|
||||
PARSED_ARGUMENTS=0
|
||||
if [[ $1 =~ [a-z] ]]; then
|
||||
BOOST_TEST_RUN_FILTERS=$1
|
||||
PARSED_ARGUMENTS=$((PARSED_ARGUMENTS + 1))
|
||||
shift
|
||||
fi
|
||||
if [[ $1 =~ ^[0-9]+$ ]]; then
|
||||
N_TEST_RUNS=$1
|
||||
PARSED_ARGUMENTS=$((PARSED_ARGUMENTS + 1))
|
||||
shift
|
||||
fi
|
||||
if [[ ${PARSED_ARGUMENTS} == 0 || $# -gt 2 || ${N_TEST_RUNS} -lt 2 ]]; then
|
||||
print_usage
|
||||
exit
|
||||
fi
|
||||
fi
|
||||
if [[ ${BOOST_TEST_RUN_FILTERS} == "" ]]; then
|
||||
BOOST_TEST_RUN_FILTERS="$(IFS=":"; echo "!${NON_DETERMINISTIC_TESTS[*]}" | sed 's/:/:!/g')"
|
||||
else
|
||||
echo "Using Boost test filter: ${BOOST_TEST_RUN_FILTERS}"
|
||||
echo
|
||||
fi
|
||||
|
||||
if ! command -v gcov > /dev/null; then
|
||||
echo "Error: gcov not installed. Exiting."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v gcovr > /dev/null; then
|
||||
echo "Error: gcovr not installed. Exiting."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -e ${TEST_DIGIBYTE_BINARY} ]]; then
|
||||
echo "Error: Executable ${TEST_DIGIBYTE_BINARY} not found. Run \"./configure --enable-lcov\" and compile."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
get_file_suffix_count() {
|
||||
find src/ -type f -name "*.$1" | wc -l
|
||||
}
|
||||
|
||||
if [[ $(get_file_suffix_count gcno) == 0 ]]; then
|
||||
echo "Error: Could not find any *.gcno files. The *.gcno files are generated by the compiler. Run \"./configure --enable-lcov\" and re-compile."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
get_covr_filename() {
|
||||
echo "gcovr.run-$1.txt"
|
||||
}
|
||||
|
||||
TEST_RUN_ID=0
|
||||
while [[ ${TEST_RUN_ID} -lt ${N_TEST_RUNS} ]]; do
|
||||
TEST_RUN_ID=$((TEST_RUN_ID + 1))
|
||||
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Measuring coverage, run #${TEST_RUN_ID} of ${N_TEST_RUNS}"
|
||||
find src/ -type f -name "*.gcda" -exec rm {} \;
|
||||
if [[ $(get_file_suffix_count gcda) != 0 ]]; then
|
||||
echo "Error: Stale *.gcda files found. Exiting."
|
||||
exit 1
|
||||
fi
|
||||
TEST_OUTPUT_TEMPFILE=$(mktemp)
|
||||
if ! BOOST_TEST_RUN_FILTERS="${BOOST_TEST_RUN_FILTERS}" ${TEST_DIGIBYTE_BINARY} > "${TEST_OUTPUT_TEMPFILE}" 2>&1; then
|
||||
cat "${TEST_OUTPUT_TEMPFILE}"
|
||||
rm "${TEST_OUTPUT_TEMPFILE}"
|
||||
exit 1
|
||||
fi
|
||||
rm "${TEST_OUTPUT_TEMPFILE}"
|
||||
if [[ $(get_file_suffix_count gcda) == 0 ]]; then
|
||||
echo "Error: Running the test suite did not create any *.gcda files. The gcda files are generated when the instrumented test programs are executed. Run \"./configure --enable-lcov\" and re-compile."
|
||||
exit 1
|
||||
fi
|
||||
GCOVR_TEMPFILE=$(mktemp)
|
||||
if ! gcovr --gcov-executable "${GCOV_EXECUTABLE}" -r src/ > "${GCOVR_TEMPFILE}"; then
|
||||
echo "Error: gcovr failed. Output written to ${GCOVR_TEMPFILE}. Exiting."
|
||||
exit 1
|
||||
fi
|
||||
GCOVR_FILENAME=$(get_covr_filename ${TEST_RUN_ID})
|
||||
mv "${GCOVR_TEMPFILE}" "${GCOVR_FILENAME}"
|
||||
if grep -E "^TOTAL *0 *0 " "${GCOVR_FILENAME}"; then
|
||||
echo "Error: Spurious gcovr output. Make sure the correct GCOV_EXECUTABLE variable is set in $0 (\"gcov\" for gcc, \"llvm-cov gcov\" for clang)."
|
||||
exit 1
|
||||
fi
|
||||
if [[ ${TEST_RUN_ID} != 1 ]]; then
|
||||
COVERAGE_DIFF=$(diff -u "$(get_covr_filename 1)" "${GCOVR_FILENAME}")
|
||||
if [[ ${COVERAGE_DIFF} != "" ]]; then
|
||||
echo
|
||||
echo "The line coverage is non-deterministic between runs. Exiting."
|
||||
echo
|
||||
echo "The test suite must be deterministic in the sense that the set of lines executed at least"
|
||||
echo "once must be identical between runs. This is a necessary condition for meaningful"
|
||||
echo "coverage measuring."
|
||||
echo
|
||||
echo "${COVERAGE_DIFF}"
|
||||
exit 1
|
||||
fi
|
||||
rm "${GCOVR_FILENAME}"
|
||||
fi
|
||||
done
|
||||
|
||||
echo
|
||||
echo "Coverage test passed: Deterministic coverage across ${N_TEST_RUNS} runs."
|
||||
exit
|
||||
215
contrib/devtools/update-translations.py
Executable file
215
contrib/devtools/update-translations.py
Executable file
|
|
@ -0,0 +1,215 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2014 Wladimir J. van der Laan
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Run this script from the root of the repository to update all translations from
|
||||
transifex.
|
||||
It will do the following automatically:
|
||||
|
||||
- fetch all translations using the tx tool
|
||||
- post-process them into valid and committable format
|
||||
- remove invalid control characters
|
||||
- remove location tags (makes diffs less noisy)
|
||||
|
||||
TODO:
|
||||
- auto-add new translations to the build system according to the translation process
|
||||
'''
|
||||
import subprocess
|
||||
import re
|
||||
import sys
|
||||
import os
|
||||
import io
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
# Name of transifex tool
|
||||
TX = 'tx'
|
||||
# Name of source language file
|
||||
SOURCE_LANG = 'digibyte_en.ts'
|
||||
# Directory with locale files
|
||||
LOCALE_DIR = 'src/qt/locale'
|
||||
# Minimum number of messages for translation to be considered at all
|
||||
MIN_NUM_MESSAGES = 10
|
||||
# Regexp to check for DigiByte addresses
|
||||
ADDRESS_REGEXP = re.compile('([13]|bc1)[a-zA-Z0-9]{30,}')
|
||||
|
||||
def check_at_repository_root():
|
||||
if not os.path.exists('.git'):
|
||||
print('No .git directory found')
|
||||
print('Execute this script at the root of the repository', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
def fetch_all_translations():
|
||||
if subprocess.call([TX, 'pull', '-f', '-a']):
|
||||
print('Error while fetching translations', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
def find_format_specifiers(s):
|
||||
'''Find all format specifiers in a string.'''
|
||||
pos = 0
|
||||
specifiers = []
|
||||
while True:
|
||||
percent = s.find('%', pos)
|
||||
if percent < 0:
|
||||
break
|
||||
specifiers.append(s[percent+1])
|
||||
pos = percent+2
|
||||
return specifiers
|
||||
|
||||
def split_format_specifiers(specifiers):
|
||||
'''Split format specifiers between numeric (Qt) and others (strprintf)'''
|
||||
numeric = []
|
||||
other = []
|
||||
for s in specifiers:
|
||||
if s in {'1','2','3','4','5','6','7','8','9'}:
|
||||
numeric.append(s)
|
||||
else:
|
||||
other.append(s)
|
||||
|
||||
# If both numeric format specifiers and "others" are used, assume we're dealing
|
||||
# with a Qt-formatted message. In the case of Qt formatting (see https://doc.qt.io/qt-5/qstring.html#arg)
|
||||
# only numeric formats are replaced at all. This means "(percentage: %1%)" is valid, without needing
|
||||
# any kind of escaping that would be necessary for strprintf. Without this, this function
|
||||
# would wrongly detect '%)' as a printf format specifier.
|
||||
if numeric:
|
||||
other = []
|
||||
|
||||
# numeric (Qt) can be present in any order, others (strprintf) must be in specified order
|
||||
return set(numeric),other
|
||||
|
||||
def sanitize_string(s):
|
||||
'''Sanitize string for printing'''
|
||||
return s.replace('\n',' ')
|
||||
|
||||
def check_format_specifiers(source, translation, errors, numerus):
|
||||
source_f = split_format_specifiers(find_format_specifiers(source))
|
||||
# assert that no source messages contain both Qt and strprintf format specifiers
|
||||
# if this fails, go change the source as this is hacky and confusing!
|
||||
assert(not(source_f[0] and source_f[1]))
|
||||
try:
|
||||
translation_f = split_format_specifiers(find_format_specifiers(translation))
|
||||
except IndexError:
|
||||
errors.append("Parse error in translation for '%s': '%s'" % (sanitize_string(source), sanitize_string(translation)))
|
||||
return False
|
||||
else:
|
||||
if source_f != translation_f:
|
||||
if numerus and source_f == (set(), ['n']) and translation_f == (set(), []) and translation.find('%') == -1:
|
||||
# Allow numerus translations to omit %n specifier (usually when it only has one possible value)
|
||||
return True
|
||||
errors.append("Mismatch between '%s' and '%s'" % (sanitize_string(source), sanitize_string(translation)))
|
||||
return False
|
||||
return True
|
||||
|
||||
def all_ts_files(suffix=''):
|
||||
for filename in os.listdir(LOCALE_DIR):
|
||||
# process only language files, and do not process source language
|
||||
if not filename.endswith('.ts'+suffix) or filename == SOURCE_LANG+suffix:
|
||||
continue
|
||||
if suffix: # remove provided suffix
|
||||
filename = filename[0:-len(suffix)]
|
||||
filepath = os.path.join(LOCALE_DIR, filename)
|
||||
yield(filename, filepath)
|
||||
|
||||
FIX_RE = re.compile(b'[\x00-\x09\x0b\x0c\x0e-\x1f]')
|
||||
def remove_invalid_characters(s):
|
||||
'''Remove invalid characters from translation string'''
|
||||
return FIX_RE.sub(b'', s)
|
||||
|
||||
# Override cdata escape function to make our output match Qt's (optional, just for cleaner diffs for
|
||||
# comparison, disable by default)
|
||||
_orig_escape_cdata = None
|
||||
def escape_cdata(text):
|
||||
text = _orig_escape_cdata(text)
|
||||
text = text.replace("'", ''')
|
||||
text = text.replace('"', '"')
|
||||
return text
|
||||
|
||||
def contains_digibyte_addr(text, errors):
|
||||
if text != None and ADDRESS_REGEXP.search(text) != None:
|
||||
errors.append('Translation "%s" contains a digibyte address. This will be removed.' % (text))
|
||||
return True
|
||||
return False
|
||||
|
||||
def postprocess_translations(reduce_diff_hacks=False):
|
||||
print('Checking and postprocessing...')
|
||||
|
||||
if reduce_diff_hacks:
|
||||
global _orig_escape_cdata
|
||||
_orig_escape_cdata = ET._escape_cdata
|
||||
ET._escape_cdata = escape_cdata
|
||||
|
||||
for (filename,filepath) in all_ts_files():
|
||||
os.rename(filepath, filepath+'.orig')
|
||||
|
||||
have_errors = False
|
||||
for (filename,filepath) in all_ts_files('.orig'):
|
||||
# pre-fixups to cope with transifex output
|
||||
parser = ET.XMLParser(encoding='utf-8') # need to override encoding because 'utf8' is not understood only 'utf-8'
|
||||
with open(filepath + '.orig', 'rb') as f:
|
||||
data = f.read()
|
||||
# remove control characters; this must be done over the entire file otherwise the XML parser will fail
|
||||
data = remove_invalid_characters(data)
|
||||
tree = ET.parse(io.BytesIO(data), parser=parser)
|
||||
|
||||
# iterate over all messages in file
|
||||
root = tree.getroot()
|
||||
for context in root.findall('context'):
|
||||
for message in context.findall('message'):
|
||||
numerus = message.get('numerus') == 'yes'
|
||||
source = message.find('source').text
|
||||
translation_node = message.find('translation')
|
||||
# pick all numerusforms
|
||||
if numerus:
|
||||
translations = [i.text for i in translation_node.findall('numerusform')]
|
||||
else:
|
||||
translations = [translation_node.text]
|
||||
|
||||
for translation in translations:
|
||||
if translation is None:
|
||||
continue
|
||||
errors = []
|
||||
valid = check_format_specifiers(source, translation, errors, numerus) and not contains_digibyte_addr(translation, errors)
|
||||
|
||||
for error in errors:
|
||||
print('%s: %s' % (filename, error))
|
||||
|
||||
if not valid: # set type to unfinished and clear string if invalid
|
||||
translation_node.clear()
|
||||
translation_node.set('type', 'unfinished')
|
||||
have_errors = True
|
||||
|
||||
# Remove location tags
|
||||
for location in message.findall('location'):
|
||||
message.remove(location)
|
||||
|
||||
# Remove entire message if it is an unfinished translation
|
||||
if translation_node.get('type') == 'unfinished':
|
||||
context.remove(message)
|
||||
|
||||
# check if document is (virtually) empty, and remove it if so
|
||||
num_messages = 0
|
||||
for context in root.findall('context'):
|
||||
for message in context.findall('message'):
|
||||
num_messages += 1
|
||||
if num_messages < MIN_NUM_MESSAGES:
|
||||
print('Removing %s, as it contains only %i messages' % (filepath, num_messages))
|
||||
continue
|
||||
|
||||
# write fixed-up tree
|
||||
# if diff reduction requested, replace some XML to 'sanitize' to qt formatting
|
||||
if reduce_diff_hacks:
|
||||
out = io.BytesIO()
|
||||
tree.write(out, encoding='utf-8')
|
||||
out = out.getvalue()
|
||||
out = out.replace(b' />', b'/>')
|
||||
with open(filepath, 'wb') as f:
|
||||
f.write(out)
|
||||
else:
|
||||
tree.write(filepath, encoding='utf-8')
|
||||
return have_errors
|
||||
|
||||
if __name__ == '__main__':
|
||||
check_at_repository_root()
|
||||
fetch_all_translations()
|
||||
postprocess_translations()
|
||||
|
||||
22
contrib/devtools/utils.py
Executable file
22
contrib/devtools/utils.py
Executable file
|
|
@ -0,0 +1,22 @@
|
|||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2021 The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Common utility functions
|
||||
'''
|
||||
import shutil
|
||||
import sys
|
||||
import os
|
||||
from typing import List
|
||||
|
||||
|
||||
def determine_wellknown_cmd(envvar, progname) -> List[str]:
|
||||
maybe_env = os.getenv(envvar)
|
||||
maybe_which = shutil.which(progname)
|
||||
if maybe_env:
|
||||
return maybe_env.split(' ') # Well-known vars are often meant to be word-split
|
||||
elif maybe_which:
|
||||
return [ maybe_which ]
|
||||
else:
|
||||
sys.exit(f"{progname} not found")
|
||||
44
contrib/devtools/utxo_snapshot.sh
Executable file
44
contrib/devtools/utxo_snapshot.sh
Executable file
|
|
@ -0,0 +1,44 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# Copyright (c) 2019 The DigiByte Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
#
|
||||
export LC_ALL=C
|
||||
|
||||
set -ueo pipefail
|
||||
|
||||
if (( $# < 3 )); then
|
||||
echo 'Usage: utxo_snapshot.sh <generate-at-height> <snapshot-out-path> <digibyte-cli-call ...>'
|
||||
echo
|
||||
echo " if <snapshot-out-path> is '-', don't produce a snapshot file but instead print the "
|
||||
echo " expected assumeutxo hash"
|
||||
echo
|
||||
echo 'Examples:'
|
||||
echo
|
||||
echo " ./contrib/devtools/utxo_snapshot.sh 570000 utxo.dat ./src/digibyte-cli -datadir=\$(pwd)/testdata"
|
||||
echo ' ./contrib/devtools/utxo_snapshot.sh 570000 - ./src/digibyte-cli'
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GENERATE_AT_HEIGHT="${1}"; shift;
|
||||
OUTPUT_PATH="${1}"; shift;
|
||||
# Most of the calls we make take a while to run, so pad with a lengthy timeout.
|
||||
DIGIBYTE_CLI_CALL="${*} -rpcclienttimeout=9999999"
|
||||
|
||||
# Block we'll invalidate/reconsider to rewind/fast-forward the chain.
|
||||
PIVOT_BLOCKHASH=$($DIGIBYTE_CLI_CALL getblockhash $(( GENERATE_AT_HEIGHT + 1 )) )
|
||||
|
||||
(>&2 echo "Rewinding chain back to height ${GENERATE_AT_HEIGHT} (by invalidating ${PIVOT_BLOCKHASH}); this may take a while")
|
||||
${DIGIBYTE_CLI_CALL} invalidateblock "${PIVOT_BLOCKHASH}"
|
||||
|
||||
if [[ "${OUTPUT_PATH}" = "-" ]]; then
|
||||
(>&2 echo "Generating txoutset info...")
|
||||
${DIGIBYTE_CLI_CALL} gettxoutsetinfo | grep hash_serialized_2 | sed 's/^.*: "\(.\+\)\+",/\1/g'
|
||||
else
|
||||
(>&2 echo "Generating UTXO snapshot...")
|
||||
${DIGIBYTE_CLI_CALL} dumptxoutset "${OUTPUT_PATH}"
|
||||
fi
|
||||
|
||||
(>&2 echo "Restoring chain to original height; this may take a while")
|
||||
${DIGIBYTE_CLI_CALL} reconsiderblock "${PIVOT_BLOCKHASH}"
|
||||
Loading…
Reference in a new issue