VextaProject/ Vexta
Vexta Vexta-CPU-Miner Vexta-Android-Wallet
Watch
1
0
Fork
You've already forked Vexta
 0
Code Issues Pull requests Projects Packages Wiki Activity Actions

Initial Vexta Core v0.1.0 release

Browse source
This commit is contained in:
Capri 2026-07-15 14:07:28 +00:00
commit 25c454329d
2481 changed files with 689289 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

48
contrib/README.md Normal file
View file

@ -0,0 +1,48 @@
Repository Tools
---------------------
### [Developer tools](/contrib/devtools) ###
Specific tools for developers working on this repository.
Additional tools, including the `github-merge.py` script, are available in the [maintainer-tools](https://github.com/digibyte-core/digibyte-maintainer-tools) repository.
### [Verify-Commits](/contrib/verify-commits) ###
Tool to verify that every merge commit was signed by a developer using the `github-merge.py` script.
### [Linearize](/contrib/linearize) ###
Construct a linear, no-fork, best version of the blockchain.
### [Qos](/contrib/qos) ###
A Linux bash script that will set up traffic control (tc) to limit the outgoing bandwidth for connections to the DigiByte network. This means one can have an always-on digibyted instance running, and another local digibyted/digibyte-qt instance which connects to this node and receives blocks from it.
### [Seeds](/contrib/seeds) ###
Utility to generate the pnSeed[] array that is compiled into the client.
Build Tools and Keys
---------------------
### Packaging ###
The [Debian](/contrib/debian) subfolder contains the copyright file.
All other packaging related files can be found in the [digibyte-core/packaging](https://github.com/digibyte-core/packaging) repository.
### [Gitian-descriptors](/contrib/gitian-descriptors) ###
Files used during the gitian build process. For more information about gitian, see the [the DigiByte Core documentation repository](https://github.com/digibyte-core/docs).
### [Builder keys](/contrib/builder-keys)
PGP keys used for signing DigiByte Core [release](/doc/release-process.md) results.
### [MacDeploy](/contrib/macdeploy) ###
Scripts and notes for Mac builds.
### [Gitian-build](/contrib/gitian-build.py) ###
Script for running full Gitian builds.
Test and Verify Tools
---------------------
### [TestGen](/contrib/testgen) ###
Utilities to generate test vectors for the data-driven DigiByte tests.
### [Verify Binaries](/contrib/verifybinaries) ###
This script attempts to download and verify the signature file SHA256SUMS.asc from digibyte.org.

27
contrib/builder-keys/README.md Normal file
View file

@ -0,0 +1,27 @@
## PGP keys of builders and Developers
The file `keys.txt` contains fingerprints of the public keys of builders and
active developers.
The associated keys are mainly used to sign git commits or the build results
of Guix builds.
The most recent version of each pgp key can be found on most pgp key servers.
Fetch the latest version from the key server to see if any key was revoked in
the meantime.
To fetch the latest version of all pgp keys in your gpg homedir,
```sh
gpg --refresh-keys
```
To fetch keys of builders and active developers, feed the list of fingerprints
of the primary keys into gpg:
```sh
while read fingerprint keyholder_name; do gpg --keyserver hkp://subset.pool.sks-keyservers.net --recv-keys ${fingerprint}; done < ./keys.txt
```
Add your key to the list if you provided Gitian signatures for two major or
minor releases of DigiByte Core.

55
contrib/builder-keys/keys.txt Normal file
View file

@ -0,0 +1,55 @@
8621501669119D1A98DE636EB9308850E0AC417E Jared Tate (JaredTate)
63250DC0970025475B4E25FF0939B74825FD3E8C GTO90 (GTO90)
33F3981290BC4570622151128ABB4093D1E79CD9 John Song (jsong)
84ADA20E9811B8327FCB37264B74E9700EDADC4D Yoshi Jaeger (SmartArray)
9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C Aaron Clauson (sipsorcery)
617C90010B3BD370B0AC7D424BB42E31C79111B8 Akira Takizawa (akx20000)
E944AE667CF960B1004BC32FCA662BE18B877A60 Andreas Schildbach (aschildbach)
152812300785C96444D3334D17565732E08E5E41 Andrew Chow (achow101)
590B7292695AFFA5B672CBB2E13FC145CD3F4304 Antoine Poinsot (darosior)
0AD83877C1F0CD1EE9BD660AD7CC770B81FD22A8 Ben Carman (benthecarman)
912FD3228387123DC97E0E57D5566241A0295FA9 BtcDrak (dgbdrak)
C519EBCF3B926298946783EFF6430754120EC2F4 Christian Decker (cdecker)
18AE2F798E0D239755DA4FD24B79F986CBDF8736 Chun Kuan Le (ken2812221)
101598DC823C1B5F9A6624ABA5E0907A0380E6C3 CoinForensics (CoinForensics)
F20F56EF6A067F70E8A5C99FFF95FAA971697405 centaur (centaur)
C060A6635913D98A3587D7DB1C2491FFEB0EF770 Cory Fields (cfields)
BF6273FAEF7CC0BA1F562E50989F6B3048A116B5 Dev Random (devrandom)
6D3170C1DC2C6FD0AEEBCA6743811D1A26623924 Douglas Roark (droark)
1C6621605EC50319C463D56C7F81D87985D61612 Emanuele Cisbani (cisba)
9A1689B60D1B3CCE9262307A2F40A9BF167FBA47 Erik Mossberg (erkmos)
D35176BE9264832E4ACA8986BF0792FBE95DC863 fivepiece (fivepiece)
6F993B250557E7B016ADE5713BDCDA2D87A881D9 Fuzzbawls (Fuzzbawls)
01CDF4627A3B88AAE4A571C87588242FBE38D3A8 Gavin Andresen (gavinandresen)
D1DBF2C4B96F2DEBF4C16654410108112E7EA81F Hennadii Stepanov (hebasto)
A2FD494D0021AA9B4FA58F759102B7AE654A4A5A Ilyas Ridhuan (IlyasRidhuan)
D3F22A3A4C366C2DCB66D3722DA9C5A7FA81EA35 Jarol Rodriguez (jarolrod)
7480909378D544EA6B6DCEB7535B12980BB8A4D3 Jeffri H Frontz (jhfrontz)
D3CC177286005BB8FF673294C5242A1AB3936517 jl2012 (jl2012)
82921A4B88FD454B7EB8CE3C796C4109063D4EAF Jon Atack (jonatack)
32EE5C4C3FA15CCADB46ABE529D4BCB6416F53EC Jonas Schnelli (jonasschnelli)
4B4E840451149DD7FB0D633477DFAB5C3108B9A8 Jorge Timon (jtimon)
C42AFF7C61B3E44A1454CD3557AF762DB3353322 Karl-Johan Alm (kallewoof)
30DE693AE0DE9E37B3E7EB6BBFF0F67810C1EED1 Lisa Neigut (niftynei)
E463A93F5F3117EEDE6C7316BD02942421F4889F Luke Dashjr (luke-jr)
B8B3F1C0E58C15DB6A81D30C3648A882F4316B9B Marco Falke (marco)
07DF3E57A548CCFB7530709189BBB8663E2E65CE Matt Corallo (BlueMatt)
CA03882CB1FC067B5D3ACFE4D300116E1C875A3D MeshCollider (meshcollider)
E777299FC265DD04793070EB944D35F9AC3DB76A Michael Ford (fanquake)
AD5764F4ADCE1B99BDFD179E12335A271D4D62EC Michael Tidwell (miketwenty1)
9692B91BBF0E8D34DFD33B1882C5C009628ECF0C Michagogo (michagogo)
C57E4B42223FDE851D4F69DD28DF2724F241D8EE midnightmagic (midnightmagic)
F4FC70F07310028424EFC20A8E4256593F177720 Oliver Gugger (guggero, Oliver Gugger)
D62A803E27E7F43486035ADBBCD04D8E9CCCAC2A Paul Rabahy (prab)
37EC7D7B0A217CDB4B4E007E7FAB114267E4FA04 Peter Todd (petertodd)
D762373D24904A3E42F33B08B9A408E71DAAC974 Pieter Wuille [Location: Leuven, Belgium] (sipa)
133EAC179436F14A5CF1B794860FEB804E669320 Pieter Wuille (sipa)
6A8F9C266528E25AEB1D7731C2371D91CB716EA7 Sebastian Falbesoner (theStack)
A8FC55F3B04BA3146F3492E79303B33A305224CB Sebastian Kung (TheCharlatan)
ED9BDF7AD6A55E232E84524257FF9BDBCC301009 Sjors Provoost (sjors)
867345026B6763E8B07EE73AB6737117397F5C4F Stephan Oeste (Emzy)
9EDAFF80E080659604F4A76B2EBB056FD847F8A7 Stephan Oeste (Emzy)
6DEEF79B050C4072509B743F8C275BC595448867 Tomas Kanocz (KanoczTomas)
AEC1884398647C47413C1C3FB1179EB7347DC10D Warren Togami (wtogami)
79D00BAC68B56D422F945A8F8E3A8F3247DBCBBF Willy Ko (willyko)
71A3B16735405025D447E8F274810B012346C9A6 Wladimir J. van der Laan (laanwj)

21
contrib/debian/README.md Normal file
View file

@ -0,0 +1,21 @@
Debian
====================
This directory contains files used to package digibyted/digibyte-qt
for Debian-based Linux systems. If you compile digibyted/digibyte-qt yourself, there are some useful files here.
## digibyte: URI support ##
digibyte-qt.desktop (Gnome / Open Desktop)
To install:
sudo desktop-file-install digibyte-qt.desktop
sudo update-desktop-database
If you build yourself, you will either need to modify the paths in
the .desktop file or copy or symlink your digibyte-qt binary to `/usr/bin`
and the `../../share/pixmaps/digibyte128.png` to `/usr/share/pixmaps`
digibyte-qt.protocol (KDE)

578
contrib/debian/changelog Normal file
View file

@ -0,0 +1,578 @@
digibyte (0.14.1-trusty4) trusty; urgency=medium
* Re-enable UPnP support.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Fri, 05 May 2017 13:28:00 -0400
digibyte (0.14.1-trusty3) trusty; urgency=medium
* Build with qt5 if we are on a non-Ubuntu (ie non-Unity) distro.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Thu, 04 May 2017 17:13:00 -0400
digibyte (0.14.1-trusty2) trusty; urgency=medium
* Bump minimum boost version in deps.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Thu, 04 May 2017 17:12:00 -0400
digibyte (0.14.1-trusty1) trusty; urgency=medium
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Sat, 22 Apr 2017 17:10:00 -0400
digibyte (0.14.0-trusty1) trusty; urgency=medium
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Wed, 08 Mar 2017 10:30:00 -0500
digibyte (0.13.2-trusty1) trusty; urgency=medium
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Thu, 05 Jan 2017 09:59:00 -0500
digibyte (0.13.1-trusty2) trusty; urgency=medium
* Revert to Qt4, due to https://github.com/digibyte/digibyte/issues/9038
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Mon, 31 Oct 2016 11:16:00 -0400
digibyte (0.13.1-trusty1) trusty; urgency=medium
* New upstream release.
* Backport updated digibyte-qt.desktop from upstream master
* Add zmq dependency
* Switch to Qt5 (breaks precise, but that was already broken by C++11)
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Thu, 27 Oct 2016 17:32:00 -0400
digibyte (0.13.0-trusty1) trusty; urgency=medium
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Sun, 04 Sep 2016 22:09:00 -0400
digibyte (0.12.1-trusty1) trusty; urgency=medium
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Mon, 18 Apr 2016 14:26:00 -0700
digibyte (0.12.0-trusty6) trusty; urgency=medium
* Fix program-options dep.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Fri, 25 Mar 2016 21:41:00 -0700
digibyte (0.12.0-trusty5) trusty; urgency=medium
* Test explicit --with-gui
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 23 Feb 2015 23:25:00 -0800
digibyte (0.12.0-trusty4) trusty; urgency=medium
* Fix libevent-dev dep.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 23 Feb 2015 23:25:00 -0800
digibyte (0.12.0-trusty3) trusty; urgency=medium
* Fix precise boost dep.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 23 Feb 2015 19:55:00 -0800
digibyte (0.12.0-trusty2) trusty; urgency=medium
* Fix libevent dep.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 23 Feb 2015 19:53:00 -0800
digibyte (0.12.0-trusty1) trusty; urgency=medium
* New upstream release
* Various updates to contrib/debian were merged, a few were not
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 23 Feb 2015 19:29:00 -0800
digibyte (0.11.2-trusty1) trusty; urgency=low
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Fri, 13 Nov 2015 18:39:00 -0800
digibyte (0.11.1-trusty2) trusty; urgency=low
* Remove minupnpc builddep.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Wed, 14 Oct 2015 23:06:00 -1000
digibyte (0.11.1-trusty1) trusty; urgency=high
* New upstream release.
* Disable all UPnP support.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Wed, 14 Oct 2015 13:57:00 -1000
digibyte (0.11.0-precise1) precise; urgency=medium
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 14 Jul 2015 14:39:00 -1000
digibyte (0.10.2-precise1) precise; urgency=medium
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Mon, 29 Jun 2015 17:33:00 -1000
digibyte (0.10.1-precise3) precise; urgency=medium
* Fix build dep (include python).
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 5 May 2015 09:28:00 -1000
digibyte (0.10.1-precise2) precise; urgency=medium
* Fix miniupnpc dep.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 5 May 2015 00:33:00 -1000
digibyte (0.10.1-precise1) precise; urgency=medium
* New upstream release.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Tue, 5 May 2015 00:07:00 -1000
digibyte (0.10.0-precise1) precise; urgency=medium
* New upstream releases.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Wed, 18 Feb 2015 13:22:00 -1000
digibyte (0.9.4-precise1) precise; urgency=high
* New upstream releases.
-- Matt Corallo (laptop - only while traveling) <matt@mattcorallo.com> Mon, 12 Jan 2015 23:30:00 -1000
digibyte (0.9.3-precise1) precise; urgency=medium
* New upstream releases.
-- Matt Corallo (BlueMatt) <matt@mattcorallo.com> Fri, 26 Sep 2014 12:01:00 -0700
digibyte (0.9.1-precise1) precise; urgency=medium
* New upstream release.
* Backport pull #4019
-- Matt Corallo <matt@bluematt.me> Sat, 19 Apr 2014 17:29:00 -0400
digibyte (0.9.0-precise1) precise; urgency=medium
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Thu, 20 Mar 2014 13:10:00 -0400
digibyte (0.8.6-precise1) precise; urgency=medium
* New upstream release.
* Make .desktop paths non-fixed (suggested by prusnak@github)
-- Matt Corallo <matt@bluematt.me> Fri, 13 Dec 2013 13:31:00 -0400
digibyte (0.8.5-precise1) precise; urgency=medium
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Sun, 15 Sep 2013 14:02:00 -0400
digibyte (0.8.4-precise1) precise; urgency=medium
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Wed, 4 Sep 2013 10:25:00 -0400
digibyte (0.8.3-natty1) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Wed, 26 Jun 2013 00:18:00 +0100
digibyte (0.8.2-natty1) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Wed, 29 Mar 2013 23:23:00 +0100
digibyte (0.8.1-natty3) natty; urgency=low
* New pixmaps
-- Jonas Schnelli <jonas.schnelli@include7.ch> Mon, 13 May 2013 16:14:00 +0100
digibyte (0.8.1-natty2) natty; urgency=low
* Remove dumb broken launcher script
-- Matt Corallo <matt@bluematt.me> Sun, 24 Mar 2013 20:01:00 -0400
digibyte (0.8.1-natty1) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Tue, 19 Mar 2013 13:03:00 -0400
digibyte (0.8.0-natty1) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Sat, 23 Feb 2013 16:01:00 -0500
digibyte (0.7.2-natty1) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Sat, 15 Dec 2012 10:59:00 -0400
digibyte (0.7.1-natty1) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Wed, 24 Oct 2012 15:06:00 -0400
digibyte (0.7.0-natty1) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Mon, 17 Sep 2012 13:45:00 +0200
digibyte (0.6.3-natty1) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Mon, 25 Jun 2012 23:47:00 +0200
digibyte (0.6.2-natty1) natty; urgency=low
* Update package description and launch scripts.
-- Matt Corallo <matt@bluematt.me> Sat, 2 Jun 2012 16:41:00 +0200
digibyte (0.6.2-natty0) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Tue, 8 May 2012 16:27:00 -0500
digibyte (0.6.1-natty0) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Sun, 6 May 2012 20:09:00 -0500
digibyte (0.6.0-natty0) natty; urgency=low
* New upstream release.
* Add GNOME/KDE support for digibyte-qt's digibyte: URI support.
Thanks to luke-jr for the KDE .protocol file.
-- Matt Corallo <matt@bluematt.me> Sat, 31 Mar 2012 15:35:00 -0500
digibyte (0.5.3-natty1) natty; urgency=low
* Mark for upload to PPA.
-- Matt Corallo <matt@bluematt.me> Wed, 14 Mar 2012 23:06:00 -0400
digibyte (0.5.3-natty0) natty; urgency=low
* New upstream release.
-- Luke Dashjr <luke+digibyte+deb@dashjr.org> Tue, 10 Jan 2012 15:57:00 -0500
digibyte (0.5.2-natty1) natty; urgency=low
* Remove mentions on anonymity in package descriptions and manpage.
These should never have been there, digibyte isnt anonymous without
a ton of work that virtually no users will ever be willing and
capable of doing
-- Matt Corallo <matt@bluematt.me> Sat, 7 Jan 2012 13:37:00 -0500
digibyte (0.5.2-natty0) natty; urgency=low
* New upstream release.
-- Luke Dashjr <luke+digibyte+deb@dashjr.org> Fri, 16 Dec 2011 17:57:00 -0500
digibyte (0.5.1-natty0) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Fri, 16 Dec 2011 13:27:00 -0500
digibyte (0.5.0-natty0) natty; urgency=low
* New upstream release.
-- Matt Corallo <matt@bluematt.me> Mon, 21 Nov 2011 11:32:00 -0500
digibyte (0.5.0~rc7-natty0) natty; urgency=low
* New upstream release candidate.
-- Matt Corallo <matt@bluematt.me> Sun, 20 Nov 2011 17:08:00 -0500
digibyte (0.5.0~rc3-natty0) natty; urgency=low
* New upstream release candidate.
* Don't set rpcpassword for digibyte-qt.
-- Matt Corallo <matt@bluematt.me> Tue, 8 Nov 2011 11:56:00 -0400
digibyte (0.5.0~rc1-natty1) natty; urgency=low
* Add test_digibyte to build test
* Fix clean
* Remove uneccessary build-dependancies
-- Matt Corallo <matt@bluematt.me> Wed, 26 Oct 2011 14:37:18 -0400
digibyte (0.5.0~rc1-natty0) natty; urgency=low
* Mark for natty
* Fix broken build
* Fix copyright listing
* Remove digibyte: URL handler until digibyte actually has support for it (Oops)
-- Matt Corallo <matt@bluematt.me> Wed, 26 Oct 2011 14:37:18 -0400
digibyte (0.5.0~rc1-2) experimental; urgency=low
* Add digibyte-qt
-- Matt Corallo <matt@bluematt.me> Tue, 25 Oct 2011 15:24:18 -0400
digibyte (0.5.0~rc1-1) experimental; urgency=low
* New upstream prerelease.
* Add Github as alternate upstream source in watch file.
* Stop build-depending on libcrypto++-dev, and drop patch 1000:
Upstream no longer use crypto++.
* Drop patch 1003: Upstream builds dynamic by default now.
* Update copyright file: Drop notes on longer included sources.
-- Jonas Smedegaard <dr@jones.dk> Fri, 14 Oct 2011 00:16:18 +0200
digibyte (0.4.0-1) unstable; urgency=low
* New upstream release.
* Stop repackaging source tarballs: No DFSG-violating stripping left.
* Update copyright file:
+ Add Github URL to Source.
* Drop dpkg-source local-options hint: Declared options are default
since dpkg-source 1.16.1.
+ Add irc URL to Upstream-Contact.
+ Add comment on DigiByte Developers to catch-all Files section.
+ Add Files sections for newly readded src/cryptopp/* (new custom
BSD-like license), and newly added doc/build-osx.txt and
src/makefile.osx (Expat).
* Bump debhelper compatibility level to 7.
* Suppress binary icns and gpg files.
* Enable regression tests:
+ Build-depend on libboost-test-dev.
+ Extend patch 1003 to also dynamically link test binary.
+ Build and invoke test binary unless tests are disabled.
* Tighten build-dependency on cdbs: Recent version needed to support
debhelper 7.
* Relax build-depend unversioned on debhelper: needed version
satisfied even in oldstable.
* Stop suppress optional build-dependencies: Satisfied in stable.
Build-depend on devscripts (enabling copyright-check).
-- Jonas Smedegaard <dr@jones.dk> Wed, 05 Oct 2011 01:48:53 +0200
digibyte (0.3.24~dfsg-1) unstable; urgency=low
* New upstream release.
[ Jonas Smedegaard ]
* Improve various usage hints:
+ Explicitly mention in long description that digibyted contains
daemon and command-line interface.
+ Extend README.Debian with section on lack of GUI, and add primary
headline.
+ Avoid installing upstream README: contains no parts relevant for
Debian usage.
Thanks to richard for suggestions (see bug#629443).
* Favor final releases over prereleases in rules and watch file.
Thanks to Jan Dittberner.
* Track -src (not -linux) tarballs in rules and watch file.
Thanks to Jan Dittberner.
* Drop patches 1004 and 1005 (integrated upstream) and simplify
CXXFLAGS in rules file.
* Stop stripping no longer included source-less binaries from upstream
tarballs.
[ Jan Dittberner ]
* refresh debian/patches/1000_use_system_crypto++.patch
-- Jonas Smedegaard <dr@jones.dk> Tue, 19 Jul 2011 15:08:54 +0200
digibyte (0.3.21~dfsg-2) unstable; urgency=low
* Enable UPNP support:
+ Drop patch 1006.
+ Build-depend on libminiupnpc-dev.
Thanks to Matt Corallo.
-- Jonas Smedegaard <dr@jones.dk> Sat, 28 May 2011 15:52:44 +0200
digibyte (0.3.21~dfsg-1) unstable; urgency=low
* New upstream release.
* Refresh patches.
* Drop patch 1002: no longer needed, as upstream use pkgconfig now.
* Add patch 1006 to really unset USE_UPNP as aparently intended.
* Adjust cleanup rule to preserve .gitignore files.
* Update copyright file:
+ Bump format to draft 174 of DEP-5.
+ Shorten comments.
* Bump policy compliance to standards-version 3.9.2.
* Shorten Vcs-Browser paragraph in control file.
* Fix mention daemon (not CLI tools) in short description.
* Stop conflicting with or replace digibyte-cli: Only transitional, no
longer needed.
* Link against unversioned berkeleydb. Update NEWS and README.Debian
accordingly (and improve wording while at it).
Closes: Bug#621425. Thanks to Ondřej Surý.
* This release also implicitly updates linkage against libcrypto++,
which closes: bug#626953, #627024.
* Disable linkage against not yet Debian packaged MiniUPnP.
* Silence seemingly harmless noise about unused variables.
-- Jonas Smedegaard <dr@jones.dk> Tue, 17 May 2011 15:31:24 +0200
digibyte (0.3.20.2~dfsg-2) unstable; urgency=medium
* Fix have wrapper script execute real binary (not loop executing
itself).
Closes: bug#617290. Thanks to Philippe Gauthier and Etienne Laurin.
* Set urgency=medium as the only (user-exposed) binary is useless
without this fix and has been for some time.
-- Jonas Smedegaard <dr@jones.dk> Wed, 16 Mar 2011 09:11:06 +0100
digibyte (0.3.20.2~dfsg-1) unstable; urgency=low
* New upstream release.
* Fix provide and replace former package name digibyte-cli.
Closes: bug#618439. Thanks to Shane Wegner.
-- Jonas Smedegaard <dr@jones.dk> Tue, 15 Mar 2011 11:41:43 +0100
digibyte (0.3.20.01~dfsg-1) unstable; urgency=low
* New upstream release.
[ Micah Anderson ]
* Add myself as uploader.
[ Jonas Smedegaard ]
* Add wrapper for digibyted to ease initial startup.
* Update patches:
+ Drop patch 2002: Applied upstream.
+ Add patch 1005 to add phtread linker option.
Closes: bug#615619. Thanks to Shane Wegner.
+ Refresh patches.
* Extend copyright years in rules file header.
* Rewrite copyright file using draft svn166 of DEP5 format.
* Rename binary package to digibyted (from bincoin-cli).
Closes: bug#614025. Thanks to Luke-Jr.
-- Jonas Smedegaard <dr@jones.dk> Tue, 01 Mar 2011 15:55:04 +0100
digibyte (0.3.19~dfsg-6) unstable; urgency=low
* Fix override agressive optimizations.
* Fix tighten build-dependencies to really fit backporting to Lenny:
+ Add fallback build-dependency on libdb4.6++-dev.
+ Tighten unversioned Boost build-dependencies to recent versions,
To force use of versioned Boost when backporting to Lenny.
...needs more love, though: actual build fails.
-- Jonas Smedegaard <dr@jones.dk> Mon, 17 Jan 2011 19:48:35 +0100
digibyte (0.3.19~dfsg-5) unstable; urgency=low
* Fix lower Boost fallback-build-dependencies to 1.35, really
available in Lenny.
* Correct comment in rules file regarding reason for versioned Boost
fallback-build-dependency.
* Add patch 2002 adding -mt decoration to Boost flags, to ease
backporting to Lenny.
* Respect DEB_BUILD_OPTIONS, and suppress arch-specific optimizations:
+ Add patch 1004 to allow overriding optimization flags.
+ Set optimization flags conditionally at build time.
+ Drop patch 2002 unconditionally suppressing arch-optimizations.
-- Jonas Smedegaard <dr@jones.dk> Mon, 17 Jan 2011 16:04:48 +0100
digibyte (0.3.19~dfsg-4) unstable; urgency=low
[ Micah Anderson ]
* Provide example digibyte.conf.
* Add digibyted(1) and digibyte.conf(5) man pages.
[ Jonas Smedegaard ]
* Ease backporting:
+ Suppress optional build-dependencies.
+ Add fallback build-dependencies on the most recent Boost libs
available in Lenny (where unversioned Boost libs are missing).
* Add Micah as copyright holder for manpages, licensed as GPL-3+.
* Bump copyright format to Subversion candidate draft 162 of DEP5.
-- Jonas Smedegaard <dr@jones.dk> Mon, 17 Jan 2011 14:00:48 +0100
digibyte (0.3.19~dfsg-3) unstable; urgency=low
* Document in copyright file files excluded from repackaged source.
* Update copyright file:
+ Bump DEP5 format hint to Subversion draft rev. 153.
+ Consistently wrap at 72 chars.
+ Refer to GPL-2 file (not GPL symlink).
* Link against Berkeley DB 4.8 (not 4.7):
+ Build-depend on libdb4.8++-dev (and on on libdb4.7++-dev).
+ Suggest libdb4.8-util and db4.7-util.
+ Add README.Debian note on (untested) upgrade routine.
+ Add NEWS entry on changed db version, referring to README.Debian.
-- Jonas Smedegaard <dr@jones.dk> Fri, 07 Jan 2011 22:50:57 +0100
digibyte (0.3.19~dfsg-2) unstable; urgency=low
* Adjust build options to use optimized miner only for amd64. Fixes
FTBFS on i386 (and other archs, if compiling anywhere else at all).
* Avoid static linking.
* Adjust patch 2001 to avoid only arch-specific optimizations (keep
-O3).
* Extend long description to mention disk consumption and initial use
of IRC.
All of above changes thanks to Helmuth Grohne.
* Add lintian override regarding OpenSSL and GPL: Linked code is Expat
- only Debian packaging is GPL-2+.
-- Jonas Smedegaard <dr@jones.dk> Wed, 29 Dec 2010 00:27:54 +0100
digibyte (0.3.19~dfsg-1) unstable; urgency=low
[ Jonas Smedegaard ]
* Initial release.
Closes: bug#578157.
-- Jonas Smedegaard <dr@jones.dk> Tue, 28 Dec 2010 15:49:22 +0100

70
contrib/debian/control Normal file
View file

@ -0,0 +1,70 @@
Source: digibyte
Section: utils
Priority: optional
Maintainer: Matt Corallo <matt@mattcorallo.com>
Uploaders: Matt Corallo <matt@mattcorallo.com>
Build-Depends: debhelper,
devscripts,
automake,
libtool,
bash-completion,
libdb4.8++-dev,
libssl-dev,
pkg-config,
libevent-dev,
libboost-system1.48-dev | libboost-system-dev (>> 1.47),
libboost-filesystem1.48-dev | libboost-filesystem-dev (>> 1.47),
libboost-program-options1.48-dev | libboost-program-options-dev (>> 1.47),
libboost-thread1.48-dev | libboost-thread-dev (>> 1.47),
libboost-test1.48-dev | libboost-test-dev (>> 1.47),
libboost-chrono1.48-dev | libboost-chrono-dev (>> 1.47),
libminiupnpc8-dev | libminiupnpc-dev,
qt4-qmake, libqt4-dev,
qttools5-dev-tools, qttools5-dev,
libqrencode-dev,
libprotobuf-dev, protobuf-compiler,
python,
libzmq3-dev
Standards-Version: 3.9.2
Homepage: https://digibyte.org/
Vcs-Git: git://github.com/digibyte/digibyte.git
Vcs-Browser: https://github.com/digibyte/digibyte
Package: digibyted
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: peer-to-peer network based digital currency - daemon
DigiByte is a free open source peer-to-peer electronic cash system that
is completely decentralized, without the need for a central server or
trusted parties. Users hold the crypto keys to their own money and
transact directly with each other, with the help of a P2P network to
check for double-spending.
.
This package provides the daemon, digibyted, and the CLI tool
digibyte-cli to interact with the daemon.
Package: digibyte-qt
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: peer-to-peer network based digital currency - Qt GUI
DigiByte is a free open source peer-to-peer electronic cash system that
is completely decentralized, without the need for a central server or
trusted parties. Users hold the crypto keys to their own money and
transact directly with each other, with the help of a P2P network to
check for double-spending.
.
This package provides DigiByte-Qt, a GUI for DigiByte based on Qt.
Package: digibyte-tx
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: peer-to-peer digital currency - standalone transaction tool
DigiByte is a free open source peer-to-peer electronic cash system that
is completely decentralized, without the need for a central server or
trusted parties. Users hold the crypto keys to their own money and
transact directly with each other, with the help of a P2P network to
check for double-spending.
.
This package provides digibyte-tx, a command-line transaction creation
tool which can be used without a digibyte daemon. Some means of
exchanging minimal transaction data with peers is still required.

161
contrib/debian/copyright Normal file
View file

@ -0,0 +1,161 @@
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: DigiByte
Upstream-Contact: Satoshi Nakamoto <satoshin@gmx.com>
irc://#digibyte-core-dev@libera.chat
Source: https://github.com/digibyte-core/digibyte
Files: *
Copyright: 2009-2023, DigiByte Core Developers
License: Expat
Comment: The DigiByte Core Developers encompasses the current developers listed on digibyte.org,
as well as the numerous contributors to the project.
Files: debian/*
Copyright: 2010-2011, Jonas Smedegaard <dr@jones.dk>
2011, Matt Corallo <matt@bluematt.me>
License: GPL-2+
Files: src/secp256k1/build-aux/m4/ax_jni_include_dir.m4
Copyright: 2008 Don Anderson <dda@sleepycat.com>
License: GNU-All-permissive-License
Files: src/secp256k1/build-aux/m4/ax_prog_cc_for_build.m4
Copyright: 2008 Paolo Bonzini <bonzini@gnu.org>
License: GNU-All-permissive-License
Files: src/qt/res/icons/add.png
src/qt/res/icons/address-book.png
src/qt/res/icons/chevron.png
src/qt/res/icons/edit.png
src/qt/res/icons/editcopy.png
src/qt/res/icons/editpaste.png
src/qt/res/icons/export.png
src/qt/res/icons/eye.png
src/qt/res/icons/history.png
src/qt/res/icons/lock_*.png
src/qt/res/icons/overview.png
src/qt/res/icons/receive.png
src/qt/res/icons/remove.png
src/qt/res/icons/send.png
src/qt/res/icons/synced.png
src/qt/res/icons/transaction*.png
src/qt/res/icons/tx_output.png
src/qt/res/icons/warning.png
Copyright: Stephen Hutchings (and more)
http://typicons.com
License: Expat
Comment: Site: https://github.com/stephenhutchings/typicons.font
Files: src/qt/res/icons/connect*.png
src/qt/res/src/connect-*.svg
src/qt/res/icons/network_disabled.png
src/qt/res/src/network_disabled.svg
Copyright: Marco Falke
Luke Dashjr
License: Expat
Comment: Inspired by Stephen Hutchings' Typicons
Files: src/qt/res/icons/tx_mined.png
src/qt/res/src/mine.svg
src/qt/res/icons/fontbigger.png
src/qt/res/icons/fontsmaller.png
src/qt/res/icons/hd_disabled.png
src/qt/res/src/hd_disabled.svg
src/qt/res/icons/hd_enabled.png
src/qt/res/src/hd_enabled.svg
Copyright: Jonas Schnelli
License: Expat
Files: src/qt/res/icons/clock*.png
src/qt/res/icons/eye_*.png
src/qt/res/icons/tx_in*.png
src/qt/res/src/clock_*.svg
src/qt/res/src/tx_*.svg
Copyright: Stephen Hutchings, Jonas Schnelli
License: Expat
Comment: Modifications of Stephen Hutchings' Typicons
Files: src/qt/res/icons/digibyte.*
share/pixmaps/digibyte*
src/qt/res/src/digibyte.svg
Copyright: Bitboy, Jonas Schnelli
License: public-domain
Comment: Site: https://digibytetalk.org/?topic=1756.0
Files: src/qt/res/icons/proxy.png
src/qt/res/src/proxy.svg
Copyright: Cristian Mircea Messel
License: public-domain
Files: src/qt/res/fonts/RobotoMono-Bold.ttf
License: Apache-2.0
Comment: Site: https://fonts.google.com/specimen/Roboto+Mono
License: Expat
Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
.
The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
License: GNU-All-permissive-License
Copying and distribution of this file, with or without modification, are
permitted in any medium without royalty provided the copyright notice
and this notice are preserved. This file is offered as-is, without any
warranty.
License: GPL-2+
This program is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 2, or (at your option) any
later version.
.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
Comment:
On Debian systems the GNU General Public License (GPL) version 2 is
located in '/usr/share/common-licenses/GPL-2'.
.
You should have received a copy of the GNU General Public License along
with this program. If not, see <http://www.gnu.org/licenses/>.
License: GPL-3+
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU General Public License, Version 3 or any
later version published by the Free Software Foundation.
Comment:
On Debian systems the GNU General Public License (GPL) version 3 is
located in '/usr/share/common-licenses/GPL-3'.
.
You should have received a copy of the GNU General Public License along
with this program. If not, see <http://www.gnu.org/licenses/>.
License: public-domain
This work is in the public domain.
License: Apache-2.0
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

14
contrib/debian/digibyte-qt.desktop Normal file
View file

@ -0,0 +1,14 @@
[Desktop Entry]
Version=1.0
Name=DigiByte Core
Comment=Connect to the DigiByte P2P Network
Comment[de]=Verbinde mit dem DigiByte peer-to-peer Netzwerk
Comment[fr]=DigiByte, monnaie virtuelle cryptographique pair à pair
Comment[tr]=DigiByte, eşten eşe kriptografik sanal para birimi
Exec=digibyte-qt %u
Terminal=false
Type=Application
Icon=digibyte128
MimeType=x-scheme-handler/digibyte;
Categories=Office;Finance;
StartupWMClass=DigiByte-qt

6
contrib/debian/digibyte-qt.install Normal file
View file

@ -0,0 +1,6 @@
usr/local/bin/digibyte-qt usr/bin
share/pixmaps/digibyte32.xpm usr/share/pixmaps
share/pixmaps/digibyte16.xpm usr/share/pixmaps
share/pixmaps/digibyte128.png usr/share/pixmaps
debian/digibyte-qt.desktop usr/share/applications
debian/digibyte-qt.protocol usr/share/kde4/services/

2
contrib/debian/digibyte-qt.lintian-overrides Normal file
View file

@ -0,0 +1,2 @@
# Linked code is Expat - only Debian packaging is GPL-2+
digibyte-qt: possible-gpl-code-linked-with-openssl

1
contrib/debian/digibyte-qt.manpages Normal file
View file

@ -0,0 +1 @@
doc/man/digibyte-qt.1

11
contrib/debian/digibyte-qt.protocol Normal file
View file

@ -0,0 +1,11 @@
[Protocol]
exec=digibyte-qt '%u'
protocol=digibyte
input=none
output=none
helper=true
listing=
reading=false
writing=false
makedir=false
deleting=false

1
contrib/debian/digibyte-tx.bash-completion Normal file
View file

@ -0,0 +1 @@
contrib/digibyte-tx.bash-completion digibyte-tx

1
contrib/debian/digibyte-tx.install Normal file
View file

@ -0,0 +1 @@
usr/local/bin/digibyte-tx usr/bin

1
contrib/debian/digibyte-tx.manpages Normal file
View file

@ -0,0 +1 @@
doc/man/digibyte-tx.1

2
contrib/debian/digibyted.bash-completion Normal file
View file

@ -0,0 +1,2 @@
contrib/digibyted.bash-completion digibyted
contrib/digibyte-cli.bash-completion digibyte-cli

1
contrib/debian/digibyted.examples Normal file
View file

@ -0,0 +1 @@
debian/examples/digibyte.conf

2
contrib/debian/digibyted.install Normal file
View file

@ -0,0 +1,2 @@
usr/local/bin/digibyted usr/bin
usr/local/bin/digibyte-cli usr/bin

2
contrib/debian/digibyted.lintian-overrides Normal file
View file

@ -0,0 +1,2 @@
# Linked code is Expat - only Debian packaging is GPL-2+
digibyted: possible-gpl-code-linked-with-openssl

2
contrib/debian/digibyted.manpages Normal file
View file

@ -0,0 +1,2 @@
doc/man/digibyted.1
doc/man/digibyte-cli.1

191
contrib/debian/examples/digibyte.conf Normal file
View file

@ -0,0 +1,191 @@
##
## digibyte.conf configuration file. Lines beginning with # are comments.
##
# Network-related settings:
# Note that if you use testnet, signet or regtest, particularly with the options
# addnode, connect, port, bind, rpcport, rpcbind or wallet, you will also
# want to read "[Sections]" further down.
# Run on the testnet network
#testnet=0
# Run on a signet network
#signet=0
# Run a regression test network
#regtest=0
# Connect via a SOCKS5 proxy
#proxy=127.0.0.1:9050
# Bind to given address and always listen on it. Use [host]:port notation for IPv6
#bind=<addr>
# Bind to given address and add permission flags to peers connecting to it. Use [host]:port notation for IPv6
#whitebind=perm@<addr>
##############################################################
## Quick Primer on addnode vs connect ##
## Let's say for instance you use addnode=4.2.2.4 ##
## addnode will connect you to and tell you about the ##
## nodes connected to 4.2.2.4. In addition it will tell ##
## the other nodes connected to it that you exist so ##
## they can connect to you. ##
## connect will not do the above when you 'connect' to it. ##
## It will *only* connect you to 4.2.2.4 and no one else.##
## ##
## So if you're behind a firewall, or have other problems ##
## finding nodes, add some using 'addnode'. ##
## ##
## If you want to stay private, use 'connect' to only ##
## connect to "trusted" nodes. ##
## ##
## If you run multiple nodes on a LAN, there's no need for ##
## all of them to open lots of connections. Instead ##
## 'connect' them all to one node that is port forwarded ##
## and has lots of connections. ##
## Thanks goes to [Noodle] on Freenode. ##
##############################################################
# Use as many addnode= settings as you like to connect to specific peers
#addnode=69.164.218.197
#addnode=10.0.0.2:8333
# Alternatively use as many connect= settings as you like to connect ONLY to specific peers
#connect=69.164.218.197
#connect=10.0.0.1:8333
# Listening mode, enabled by default except when 'connect' is being used
#listen=1
# Port on which to listen for connections (default: 8333, testnet: 18333, signet: 38333, regtest: 18444)
#port=
# Maximum number of inbound + outbound connections (default: 125). This option
# applies only if inbound connections are enabled; otherwise, the number of connections
# will not be more than 11: 8 full-relay connections, 2 block-relay-only ones, and
# occasionally 1 short-lived feeler or extra outbound block-relay-only connection.
# These limits do not apply to connections added manually with the -addnode
# configuration option or the addnode RPC, which have a separate limit of 8 connections.
#maxconnections=
# Maximum upload bandwidth target in MiB per day (e.g. 'maxuploadtarget=1024' is 1 GiB per day).
# This limits the upload bandwidth for those with bandwidth limits. 0 = no limit (default: 0).
# -maxuploadtarget does not apply to peers with 'download' permission.
# For more information on reducing bandwidth utilization, see: doc/reduce-traffic.md.
#maxuploadtarget=
#
# JSON-RPC options (for controlling a running DigiByte/digibyted process)
#
# server=1 tells DigiByte-Qt and digibyted to accept JSON-RPC commands
#server=0
# Bind to given address to listen for JSON-RPC connections.
# Refer to the manpage or digibyted -help for further details.
#rpcbind=<addr>
# If no rpcpassword is set, rpc cookie auth is sought. The default `-rpccookiefile` name
# is .cookie and found in the `-datadir` being used for digibyted. This option is typically used
# when the server and client are run as the same user.
#
# If not, you must set rpcuser and rpcpassword to secure the JSON-RPC API.
#
# The config option `rpcauth` can be added to server startup argument. It is set at initialization time
# using the output from the script in share/rpcauth/rpcauth.py after providing a username:
#
# ./share/rpcauth/rpcauth.py alice
# String to be appended to digibyte.conf:
# rpcauth=alice:f7efda5c189b999524f151318c0c86$d5b51b3beffbc02b724e5d095828e0bc8b2456e9ac8757ae3211a5d9b16a22ae
# Your password:
# DONT_USE_THIS_YOU_WILL_GET_ROBBED_8ak1gI25KFTvjovL3gAM967mies3E=
#
# On client-side, you add the normal user/password pair to send commands:
#rpcuser=alice
#rpcpassword=DONT_USE_THIS_YOU_WILL_GET_ROBBED_8ak1gI25KFTvjovL3gAM967mies3E=
#
# You can even add multiple entries of these to the server conf file, and client can use any of them:
# rpcauth=bob:b2dd077cb54591a2f3139e69a897ac$4e71f08d48b4347cf8eff3815c0e25ae2e9a4340474079f55705f40574f4ec99
# How many seconds digibyte will wait for a complete RPC HTTP request.
# after the HTTP connection is established.
#rpcclienttimeout=30
# By default, only RPC connections from localhost are allowed.
# Specify as many rpcallowip= settings as you like to allow connections from other hosts,
# either as a single IPv4/IPv6 or with a subnet specification.
# NOTE: opening up the RPC port to hosts outside your local trusted network is NOT RECOMMENDED,
# because the rpcpassword is transmitted over the network unencrypted.
# server=1 tells DigiByte-Qt to accept JSON-RPC commands.
# it is also read by digibyted to determine if RPC should be enabled
#rpcallowip=10.1.1.34/255.255.255.0
#rpcallowip=1.2.3.4/24
#rpcallowip=2001:db8:85a3:0:0:8a2e:370:7334/96
# Listen for RPC connections on this TCP port:
#rpcport=14022
# You can use DigiByte or digibyted to send commands to DigiByte/digibyted
# running on another host using this option:
#rpcconnect=127.0.0.1
# Wallet options
# Specify where to find wallet, lockfile and logs. If not present, those files will be
# created as new.
#wallet=</path/to/dir>
# Create transactions that have enough fees so they are likely to begin confirmation within n blocks (default: 6).
# This setting is over-ridden by the -paytxfee option.
#txconfirmtarget=n
# Pay a transaction fee every time you send digibytes.
#paytxfee=0.000x
# Miscellaneous options
# Pre-generate this many public/private key pairs, so wallet backups will be valid for
# both prior transactions and several dozen future transactions.
#keypool=100
# Maintain coinstats index used by the gettxoutsetinfo RPC (default: 0).
#coinstatsindex=1
# Enable pruning to reduce storage requirements by deleting old blocks.
# This mode is incompatible with -txindex, -coinstatsindex and -rescan.
# 0 = default (no pruning).
# 1 = allows manual pruning via RPC.
# >=550 = target to stay under in MiB.
#prune=550
# User interface options
# Start DigiByte minimized
#min=1
# Minimize to the system tray
#minimizetotray=1
# [Sections]
# Most options apply to mainnet, testnet, signet and regtest.
# If you want to confine an option to just one network, you should add it in the
# relevant section below.
# EXCEPTIONS: The options addnode, connect, port, bind, rpcport, rpcbind and wallet
# only apply to mainnet unless they appear in the appropriate section below.
# Options only for mainnet
[main]
# Options only for testnet
[test]
# Options only for signet
[signet]
# Options only for regtest
[regtest]

23
contrib/debian/rules Executable file
View file

@ -0,0 +1,23 @@
#!/usr/bin/make -f
# -*- mode: makefile; coding: utf-8 -*-
#DEB_MAKE_CHECK_TARGET = test_digibyte
#build/digibyted::
# $(if $(filter nocheck,$(DEB_BUILD_OPTIONS)),,src/test_digibyte)
%:
dh --with bash-completion $@
override_dh_auto_clean:
if [ -f Makefile ]; then $(MAKE) distclean; fi
rm -rf Makefile.in aclocal.m4 configure src/Makefile.in src/digibyte-config.h.in src/build-aux src/qt/Makefile.in src/qt/test/Makefile.in src/test/Makefile.in
QT=$(shell dpkg-vendor --derives-from Ubuntu && echo qt4 || echo qt5)
# Yea, autogen should be run on the source archive, but I like doing git archive
override_dh_auto_configure:
./autogen.sh
./configure --with-gui=$(QT)
override_dh_auto_test:
make check

5
contrib/debian/watch Normal file
View file

@ -0,0 +1,5 @@
# Run the "uscan" command to check for upstream updates and more.
version=3
# use qa.debian.org redirector; see man uscan
opts=uversionmangle=s/(\d)(alpha|beta|rc)/$1~$2/,dversionmangle=s/~dfsg\d*// \
http://githubredir.debian.net/github/digibyte/digibyte v(.*).tar.gz

180
contrib/devtools/README.md Normal file
View file

@ -0,0 +1,180 @@
Contents
========
This directory contains tools for developers working on this repository.
clang-format-diff.py
===================
A script to format unified git diffs according to [.clang-format](../../src/.clang-format).
Requires `clang-format`, installed e.g. via `brew install clang-format` on macOS,
or `sudo apt install clang-format` on Debian/Ubuntu.
For instance, to format the last commit with 0 lines of context,
the script should be called from the git root folder as follows.
```
git diff -U0 HEAD~1.. | ./contrib/devtools/clang-format-diff.py -p1 -i -v
```
copyright\_header.py
====================
Provides utilities for managing copyright headers of `The DigiByte Core
developers` in repository source files. It has three subcommands:
```
$ ./copyright_header.py report <base_directory> [verbose]
$ ./copyright_header.py update <base_directory>
$ ./copyright_header.py insert <file>
```
Running these subcommands without arguments displays a usage string.
copyright\_header.py report \<base\_directory\> [verbose]
---------------------------------------------------------
Produces a report of all copyright header notices found inside the source files
of a repository. Useful to quickly visualize the state of the headers.
Specifying `verbose` will list the full filenames of files of each category.
copyright\_header.py update \<base\_directory\> [verbose]
---------------------------------------------------------
Updates all the copyright headers of `The DigiByte Core developers` which were
changed in a year more recent than is listed. For example:
```
// Copyright (c) <firstYear>-<lastYear> The DigiByte Core developers
```
will be updated to:
```
// Copyright (c) <firstYear>-<lastModifiedYear> The DigiByte Core developers
```
where `<lastModifiedYear>` is obtained from the `git log` history.
This subcommand also handles copyright headers that have only a single year. In
those cases:
```
// Copyright (c) <year> The DigiByte Core developers
```
will be updated to:
```
// Copyright (c) <year>-<lastModifiedYear> The DigiByte Core developers
```
where the update is appropriate.
copyright\_header.py insert \<file\>
------------------------------------
Inserts a copyright header for `The DigiByte Core developers` at the top of the
file in either Python or C++ style as determined by the file extension. If the
file is a Python file and it has `#!` starting the first line, the header is
inserted in the line below it.
The copyright dates will be set to be `<year_introduced>-<current_year>` where
`<year_introduced>` is according to the `git log` history. If
`<year_introduced>` is equal to `<current_year>`, it will be set as a single
year rather than two hyphenated years.
If the file already has a copyright for `The DigiByte Core developers`, the
script will exit.
gen-manpages.sh
===============
A small script to automatically create manpages in ../../doc/man by running the release binaries with the -help option.
This requires help2man which can be found at: https://www.gnu.org/software/help2man/
With in-tree builds this tool can be run from any directory within the
repostitory. To use this tool with out-of-tree builds set `BUILDDIR`. For
example:
```bash
BUILDDIR=$PWD/build contrib/devtools/gen-manpages.sh
```
github-merge.py
===============
A small script to automate merging pull-requests securely and sign them with GPG.
For example:
./github-merge.py 3077
(in any git repository) will help you merge pull request #3077 for the
digibyte/digibyte repository.
What it does:
* Fetch master and the pull request.
* Locally construct a merge commit.
* Show the diff that merge results in.
* Ask you to verify the resulting source tree (so you can do a make
check or whatever).
* Ask you whether to GPG sign the merge commit.
* Ask you whether to push the result upstream.
This means that there are no potential race conditions (where a
pullreq gets updated while you're reviewing it, but before you click
merge), and when using GPG signatures, that even a compromised GitHub
couldn't mess with the sources.
Setup
---------
Configuring the github-merge tool for the digibyte repository is done in the following way:
git config githubmerge.repository digibyte/digibyte
git config githubmerge.testcmd "make -j4 check" (adapt to whatever you want to use for testing)
git config --global user.signingkey mykeyid (if you want to GPG sign)
Create and verify timestamps of merge commits
---------------------------------------------
To create or verify timestamps on the merge commits, install the OpenTimestamps
client via `pip3 install opentimestamps-client`. Then, dowload the gpg wrapper
`ots-git-gpg-wrapper.sh` and set it as git's `gpg.program`. See
[the ots git integration documentation](https://github.com/opentimestamps/opentimestamps-client/blob/master/doc/git-integration.md#usage)
for further details.
optimize-pngs.py
================
A script to optimize png files in the digibyte
repository (requires pngcrush).
=======
security-check.py and test-security-check.py
============================================
Perform basic security checks on a series of executables.
symbol-check.py
===============
A script to check that the executables produced by gitian only contain
certain symbols and are only linked against allowed libraries.
For Linux this means checking for allowed gcc, glibc and libstdc++ version symbols.
This makes sure they are still compatible with the minimum supported distribution versions.
For macOS and Windows we check that the executables are only linked against libraries we allow.
Example usage after a gitian build:
find ../gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
If no errors occur the return value will be 0 and the output will be empty.
If there are any errors the return value will be 1 and output like this will be printed:
.../64/test_digibyte: symbol memcpy from unsupported version GLIBC_2.14
.../64/test_digibyte: symbol __fdelt_chk from unsupported version GLIBC_2.15
.../64/test_digibyte: symbol std::out_of_range::~out_of_range() from unsupported version GLIBCXX_3.4.15
.../64/test_digibyte: symbol _ZNSt8__detail15_List_nod from unsupported version GLIBCXX_3.4.15
circular-dependencies.py
========================
Run this script from the root of the source tree (`src/`) to find circular dependencies in the source code.
This looks only at which files include other files, treating the `.cpp` and `.h` file as one unit.
Example usage:
cd .../src
../contrib/devtools/circular-dependencies.py {*,*/*,*/*/*}.{h,cpp}

46
contrib/devtools/check-doc.py Executable file
View file

@ -0,0 +1,46 @@
#!/usr/bin/env python
# Copyright (c) 2015-2017 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
This checks if all command line args are documented.
Return value is 0 to indicate no error.
Author: @MarcoFalke
'''
from subprocess import check_output
import re
import sys
FOLDER_GREP = 'src'
FOLDER_TEST = 'src/test/'
CMD_ROOT_DIR = '`git rev-parse --show-toplevel`/%s' % FOLDER_GREP
CMD_GREP_ARGS = r"egrep -r -I '(map(Multi)?Args(\.count\(|\[)|Get(Bool)?Arg\()\"\-[^\"]+?\"' %s | grep -v '%s'" % (CMD_ROOT_DIR, FOLDER_TEST)
CMD_GREP_DOCS = r"egrep -r -I 'HelpMessageOpt\(\"\-[^\"=]+?(=|\")' %s" % (CMD_ROOT_DIR)
REGEX_ARG = re.compile(r'(?:map(?:Multi)?Args(?:\.count\(|\[)|Get(?:Bool)?Arg\()\"(\-[^\"]+?)\"')
REGEX_DOC = re.compile(r'HelpMessageOpt\(\"(\-[^\"=]+?)(?:=|\")')
# list unsupported, deprecated and duplicate args as they need no documentation
SET_DOC_OPTIONAL = set(['-rpcssl', '-benchmark', '-h', '-help', '-socks', '-tor', '-debugnet', '-whitelistalwaysrelay', '-prematurewitness', '-walletprematurewitness', '-promiscuousmempoolflags', '-blockminsize', '-dbcrashratio', '-forcecompactdb', '-usehd'])
def main():
used = check_output(CMD_GREP_ARGS, shell=True)
docd = check_output(CMD_GREP_DOCS, shell=True)
args_used = set(re.findall(REGEX_ARG,used))
args_docd = set(re.findall(REGEX_DOC,docd)).union(SET_DOC_OPTIONAL)
args_need_doc = args_used.difference(args_docd)
args_unknown = args_docd.difference(args_used)
print "Args used : %s" % len(args_used)
print "Args documented : %s" % len(args_docd)
print "Args undocumented: %s" % len(args_need_doc)
print args_need_doc
print "Args unknown : %s" % len(args_unknown)
print args_unknown
sys.exit(len(args_need_doc))
if __name__ == "__main__":
main()

92
contrib/devtools/circular-dependencies.py Executable file
View file

@ -0,0 +1,92 @@
#!/usr/bin/env python3
# Copyright (c) 2018-2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
import sys
import re
from typing import Dict, List, Set
MAPPING = {
'core_read.cpp': 'core_io.cpp',
'core_write.cpp': 'core_io.cpp',
}
# Directories with header-based modules, where the assumption that .cpp files
# define functions and variables declared in corresponding .h files is
# incorrect.
HEADER_MODULE_PATHS = [
'interfaces/'
]
def module_name(path):
if path in MAPPING:
path = MAPPING[path]
if any(path.startswith(dirpath) for dirpath in HEADER_MODULE_PATHS):
return path
if path.endswith(".h"):
return path[:-2]
if path.endswith(".c"):
return path[:-2]
if path.endswith(".cpp"):
return path[:-4]
return None
files = dict()
deps: Dict[str, Set[str]] = dict()
RE = re.compile("^#include <(.*)>")
# Iterate over files, and create list of modules
for arg in sys.argv[1:]:
module = module_name(arg)
if module is None:
print("Ignoring file %s (does not constitute module)\n" % arg)
else:
files[arg] = module
deps[module] = set()
# Iterate again, and build list of direct dependencies for each module
# TODO: implement support for multiple include directories
for arg in sorted(files.keys()):
module = files[arg]
with open(arg, 'r', encoding="utf8") as f:
for line in f:
match = RE.match(line)
if match:
include = match.group(1)
included_module = module_name(include)
if included_module is not None and included_module in deps and included_module != module:
deps[module].add(included_module)
# Loop to find the shortest (remaining) circular dependency
have_cycle: bool = False
while True:
shortest_cycle = None
for module in sorted(deps.keys()):
# Build the transitive closure of dependencies of module
closure: Dict[str, List[str]] = dict()
for dep in deps[module]:
closure[dep] = []
while True:
old_size = len(closure)
old_closure_keys = sorted(closure.keys())
for src in old_closure_keys:
for dep in deps[src]:
if dep not in closure:
closure[dep] = closure[src] + [src]
if len(closure) == old_size:
break
# If module is in its own transitive closure, it's a circular dependency; check if it is the shortest
if module in closure and (shortest_cycle is None or len(closure[module]) + 1 < len(shortest_cycle)):
shortest_cycle = [module] + closure[module]
if shortest_cycle is None:
break
# We have the shortest circular dependency; report it
module = shortest_cycle[0]
print("Circular dependency: %s" % (" -> ".join(shortest_cycle + [module])))
# And then break the dependency to avoid repeating in other cycles
deps[shortest_cycle[-1]] = deps[shortest_cycle[-1]] - set([module])
have_cycle = True
sys.exit(1 if have_cycle else 0)

166
contrib/devtools/clang-format-diff.py Executable file
View file

@ -0,0 +1,166 @@
#!/usr/bin/env python3
#
#===- clang-format-diff.py - ClangFormat Diff Reformatter ----*- python -*--===#
#
# The LLVM Compiler Infrastructure
#
# This file is distributed under the University of Illinois Open Source
# License.
#
# ============================================================
#
# University of Illinois/NCSA
# Open Source License
#
# Copyright (c) 2007-2015 University of Illinois at Urbana-Champaign.
# All rights reserved.
#
# Developed by:
#
# LLVM Team
#
# University of Illinois at Urbana-Champaign
#
# http://llvm.org
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal with
# the Software without restriction, including without limitation the rights to
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
# of the Software, and to permit persons to whom the Software is furnished to do
# so, subject to the following conditions:
#
# * Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimers.
#
# * Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimers in the
# documentation and/or other materials provided with the distribution.
#
# * Neither the names of the LLVM Team, University of Illinois at
# Urbana-Champaign, nor the names of its contributors may be used to
# endorse or promote products derived from this Software without specific
# prior written permission.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE
# SOFTWARE.
#
# ============================================================
#
#===------------------------------------------------------------------------===#
r"""
ClangFormat Diff Reformatter
============================
This script reads input from a unified diff and reformats all the changed
lines. This is useful to reformat all the lines touched by a specific patch.
Example usage for git/svn users:
git diff -U0 HEAD^ | clang-format-diff.py -p1 -i
svn diff --diff-cmd=diff -x-U0 | clang-format-diff.py -i
"""
import argparse
import difflib
import io
import re
import subprocess
import sys
# Change this to the full path if clang-format is not on the path.
binary = 'clang-format'
def main():
parser = argparse.ArgumentParser(description=
'Reformat changed lines in diff. Without -i '
'option just output the diff that would be '
'introduced.')
parser.add_argument('-i', action='store_true', default=False,
help='apply edits to files instead of displaying a diff')
parser.add_argument('-p', metavar='NUM', default=0,
help='strip the smallest prefix containing P slashes')
parser.add_argument('-regex', metavar='PATTERN', default=None,
help='custom pattern selecting file paths to reformat '
'(case sensitive, overrides -iregex)')
parser.add_argument('-iregex', metavar='PATTERN', default=
r'.*\.(cpp|cc|c\+\+|cxx|c|cl|h|hpp|m|mm|inc|js|ts|proto'
r'|protodevel|java)',
help='custom pattern selecting file paths to reformat '
'(case insensitive, overridden by -regex)')
parser.add_argument('-sort-includes', action='store_true', default=False,
help='let clang-format sort include blocks')
parser.add_argument('-v', '--verbose', action='store_true',
help='be more verbose, ineffective without -i')
args = parser.parse_args()
# Extract changed lines for each file.
filename = None
lines_by_file = {}
for line in sys.stdin:
match = re.search(r'^\+\+\+\ (.*?/){%s}(\S*)' % args.p, line)
if match:
filename = match.group(2)
if filename is None:
continue
if args.regex is not None:
if not re.match('^%s$' % args.regex, filename):
continue
else:
if not re.match('^%s$' % args.iregex, filename, re.IGNORECASE):
continue
match = re.search(r'^@@.*\+(\d+)(,(\d+))?', line)
if match:
start_line = int(match.group(1))
line_count = 1
if match.group(3):
line_count = int(match.group(3))
if line_count == 0:
continue
end_line = start_line + line_count - 1
lines_by_file.setdefault(filename, []).extend(
['-lines', str(start_line) + ':' + str(end_line)])
# Reformat files containing changes in place.
for filename, lines in lines_by_file.items():
if args.i and args.verbose:
print('Formatting {}'.format(filename))
command = [binary, filename]
if args.i:
command.append('-i')
if args.sort_includes:
command.append('-sort-includes')
command.extend(lines)
command.extend(['-style=file', '-fallback-style=none'])
p = subprocess.Popen(command,
stdout=subprocess.PIPE,
stderr=None,
stdin=subprocess.PIPE,
universal_newlines=True)
stdout, stderr = p.communicate()
if p.returncode != 0:
sys.exit(p.returncode)
if not args.i:
with open(filename, encoding="utf8") as f:
code = f.readlines()
formatted_code = io.StringIO(stdout).readlines()
diff = difflib.unified_diff(code, formatted_code,
filename, filename,
'(before formatting)', '(after formatting)')
diff_string = ''.join(diff)
if len(diff_string) > 0:
sys.stdout.write(diff_string)
if __name__ == '__main__':
main()

618
contrib/devtools/copyright_header.py Executable file
View file

@ -0,0 +1,618 @@
#!/usr/bin/env python3
# Copyright (c) 2016-2018 The Bitcoin Core developers
# Copyright (c) 2016-2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
import re
import fnmatch
import sys
import subprocess
import datetime
import os
################################################################################
# file filtering
################################################################################
EXCLUDE = [
# auto generated:
'src/qt/digibytestrings.cpp',
'src/chainparamsseeds.h',
# other external copyrights:
'src/reverse_iterator.h',
'src/test/fuzz/FuzzedDataProvider.h',
'src/tinyformat.h',
'src/bench/nanobench.h',
'test/functional/test_framework/bignum.py',
# python init:
'*__init__.py',
]
EXCLUDE_COMPILED = re.compile('|'.join([fnmatch.translate(m) for m in EXCLUDE]))
EXCLUDE_DIRS = [
# git subtrees
"src/crypto/ctaes/",
"src/leveldb/",
"src/secp256k1/",
"src/univalue/",
"src/crc32c/",
]
INCLUDE = ['*.h', '*.cpp', '*.cc', '*.c', '*.mm', '*.py', '*.sh', '*.bash-completion']
INCLUDE_COMPILED = re.compile('|'.join([fnmatch.translate(m) for m in INCLUDE]))
def applies_to_file(filename):
for excluded_dir in EXCLUDE_DIRS:
if filename.startswith(excluded_dir):
return False
return ((EXCLUDE_COMPILED.match(filename) is None) and
(INCLUDE_COMPILED.match(filename) is not None))
################################################################################
# obtain list of files in repo according to INCLUDE and EXCLUDE
################################################################################
GIT_LS_CMD = 'git ls-files --full-name'.split(' ')
GIT_TOPLEVEL_CMD = 'git rev-parse --show-toplevel'.split(' ')
def call_git_ls(base_directory):
out = subprocess.check_output([*GIT_LS_CMD, base_directory])
return [f for f in out.decode("utf-8").split('\n') if f != '']
def call_git_toplevel():
"Returns the absolute path to the project root"
return subprocess.check_output(GIT_TOPLEVEL_CMD).strip().decode("utf-8")
def get_filenames_to_examine(base_directory):
"Returns an array of absolute paths to any project files in the base_directory that pass the include/exclude filters"
root = call_git_toplevel()
filenames = call_git_ls(base_directory)
return sorted([os.path.join(root, filename) for filename in filenames if
applies_to_file(filename)])
################################################################################
# define and compile regexes for the patterns we are looking for
################################################################################
COPYRIGHT_WITH_C = r'Copyright \(c\)'
COPYRIGHT_WITHOUT_C = 'Copyright'
ANY_COPYRIGHT_STYLE = '(%s|%s)' % (COPYRIGHT_WITH_C, COPYRIGHT_WITHOUT_C)
YEAR = "20[0-9][0-9]"
YEAR_RANGE = '(%s)(-%s)?' % (YEAR, YEAR)
YEAR_LIST = '(%s)(, %s)+' % (YEAR, YEAR)
ANY_YEAR_STYLE = '(%s|%s)' % (YEAR_RANGE, YEAR_LIST)
ANY_COPYRIGHT_STYLE_OR_YEAR_STYLE = ("%s %s" % (ANY_COPYRIGHT_STYLE,
ANY_YEAR_STYLE))
ANY_COPYRIGHT_COMPILED = re.compile(ANY_COPYRIGHT_STYLE_OR_YEAR_STYLE)
def compile_copyright_regex(copyright_style, year_style, name):
return re.compile(r'%s %s,? %s( +\*)?\n' % (copyright_style, year_style, name))
EXPECTED_HOLDER_NAMES = [
"Satoshi Nakamoto\n",
"The DigiByte Core developers\n",
"The DigiByte Core developers \n",
"DigiByte Core Developers\n",
"the DigiByte Core developers\n",
"The DigiByte developers\n",
"The LevelDB Authors\. All rights reserved\.\n",
"BitPay Inc\.\n",
"BitPay, Inc\.\n",
"University of Illinois at Urbana-Champaign\.\n",
"MarcoFalke\n",
"Pieter Wuille\n",
"Pieter Wuille +\*\n",
"Pieter Wuille, Gregory Maxwell +\*\n",
"Pieter Wuille, Andrew Poelstra +\*\n",
"Andrew Poelstra +\*\n",
"Wladimir J. van der Laan\n",
"Jeff Garzik\n",
"Diederik Huys, Pieter Wuille +\*\n",
"Thomas Daede, Cory Fields +\*\n",
"Jan-Klaas Kollhof\n",
"Sam Rushing\n",
"ArtForz -- public domain half-a-node\n",
]
DOMINANT_STYLE_COMPILED = {}
YEAR_LIST_STYLE_COMPILED = {}
WITHOUT_C_STYLE_COMPILED = {}
for holder_name in EXPECTED_HOLDER_NAMES:
DOMINANT_STYLE_COMPILED[holder_name] = (
compile_copyright_regex(COPYRIGHT_WITH_C, YEAR_RANGE, holder_name))
YEAR_LIST_STYLE_COMPILED[holder_name] = (
compile_copyright_regex(COPYRIGHT_WITH_C, YEAR_LIST, holder_name))
WITHOUT_C_STYLE_COMPILED[holder_name] = (
compile_copyright_regex(COPYRIGHT_WITHOUT_C, ANY_YEAR_STYLE,
holder_name))
################################################################################
# search file contents for copyright message of particular category
################################################################################
def get_count_of_copyrights_of_any_style_any_holder(contents):
return len(ANY_COPYRIGHT_COMPILED.findall(contents))
def file_has_dominant_style_copyright_for_holder(contents, holder_name):
match = DOMINANT_STYLE_COMPILED[holder_name].search(contents)
return match is not None
def file_has_year_list_style_copyright_for_holder(contents, holder_name):
match = YEAR_LIST_STYLE_COMPILED[holder_name].search(contents)
return match is not None
def file_has_without_c_style_copyright_for_holder(contents, holder_name):
match = WITHOUT_C_STYLE_COMPILED[holder_name].search(contents)
return match is not None
################################################################################
# get file info
################################################################################
def read_file(filename):
return open(filename, 'r', encoding="utf8").read()
def gather_file_info(filename):
info = {}
info['filename'] = filename
c = read_file(filename)
info['contents'] = c
info['all_copyrights'] = get_count_of_copyrights_of_any_style_any_holder(c)
info['classified_copyrights'] = 0
info['dominant_style'] = {}
info['year_list_style'] = {}
info['without_c_style'] = {}
for holder_name in EXPECTED_HOLDER_NAMES:
has_dominant_style = (
file_has_dominant_style_copyright_for_holder(c, holder_name))
has_year_list_style = (
file_has_year_list_style_copyright_for_holder(c, holder_name))
has_without_c_style = (
file_has_without_c_style_copyright_for_holder(c, holder_name))
info['dominant_style'][holder_name] = has_dominant_style
info['year_list_style'][holder_name] = has_year_list_style
info['without_c_style'][holder_name] = has_without_c_style
if has_dominant_style or has_year_list_style or has_without_c_style:
info['classified_copyrights'] = info['classified_copyrights'] + 1
return info
################################################################################
# report execution
################################################################################
SEPARATOR = '-'.join(['' for _ in range(80)])
def print_filenames(filenames, verbose):
if not verbose:
return
for filename in filenames:
print("\t%s" % filename)
def print_report(file_infos, verbose):
print(SEPARATOR)
examined = [i['filename'] for i in file_infos]
print("%d files examined according to INCLUDE and EXCLUDE fnmatch rules" %
len(examined))
print_filenames(examined, verbose)
print(SEPARATOR)
print('')
zero_copyrights = [i['filename'] for i in file_infos if
i['all_copyrights'] == 0]
print("%4d with zero copyrights" % len(zero_copyrights))
print_filenames(zero_copyrights, verbose)
one_copyright = [i['filename'] for i in file_infos if
i['all_copyrights'] == 1]
print("%4d with one copyright" % len(one_copyright))
print_filenames(one_copyright, verbose)
two_copyrights = [i['filename'] for i in file_infos if
i['all_copyrights'] == 2]
print("%4d with two copyrights" % len(two_copyrights))
print_filenames(two_copyrights, verbose)
three_copyrights = [i['filename'] for i in file_infos if
i['all_copyrights'] == 3]
print("%4d with three copyrights" % len(three_copyrights))
print_filenames(three_copyrights, verbose)
four_or_more_copyrights = [i['filename'] for i in file_infos if
i['all_copyrights'] >= 4]
print("%4d with four or more copyrights" % len(four_or_more_copyrights))
print_filenames(four_or_more_copyrights, verbose)
print('')
print(SEPARATOR)
print('Copyrights with dominant style:\ne.g. "Copyright (c)" and '
'"<year>" or "<startYear>-<endYear>":\n')
for holder_name in EXPECTED_HOLDER_NAMES:
dominant_style = [i['filename'] for i in file_infos if
i['dominant_style'][holder_name]]
if len(dominant_style) > 0:
print("%4d with '%s'" % (len(dominant_style),
holder_name.replace('\n', '\\n')))
print_filenames(dominant_style, verbose)
print('')
print(SEPARATOR)
print('Copyrights with year list style:\ne.g. "Copyright (c)" and '
'"<year1>, <year2>, ...":\n')
for holder_name in EXPECTED_HOLDER_NAMES:
year_list_style = [i['filename'] for i in file_infos if
i['year_list_style'][holder_name]]
if len(year_list_style) > 0:
print("%4d with '%s'" % (len(year_list_style),
holder_name.replace('\n', '\\n')))
print_filenames(year_list_style, verbose)
print('')
print(SEPARATOR)
print('Copyrights with no "(c)" style:\ne.g. "Copyright" and "<year>" or '
'"<startYear>-<endYear>":\n')
for holder_name in EXPECTED_HOLDER_NAMES:
without_c_style = [i['filename'] for i in file_infos if
i['without_c_style'][holder_name]]
if len(without_c_style) > 0:
print("%4d with '%s'" % (len(without_c_style),
holder_name.replace('\n', '\\n')))
print_filenames(without_c_style, verbose)
print('')
print(SEPARATOR)
unclassified_copyrights = [i['filename'] for i in file_infos if
i['classified_copyrights'] < i['all_copyrights']]
print("%d with unexpected copyright holder names" %
len(unclassified_copyrights))
print_filenames(unclassified_copyrights, verbose)
print(SEPARATOR)
def exec_report(base_directory, verbose):
filenames = get_filenames_to_examine(base_directory)
file_infos = [gather_file_info(f) for f in filenames]
print_report(file_infos, verbose)
################################################################################
# report cmd
################################################################################
REPORT_USAGE = """
Produces a report of all copyright header notices found inside the source files
of a repository.
Usage:
$ ./copyright_header.py report <base_directory> [verbose]
Arguments:
<base_directory> - The base directory of a digibyte source code repository.
[verbose] - Includes a list of every file of each subcategory in the report.
"""
def report_cmd(argv):
if len(argv) == 2:
sys.exit(REPORT_USAGE)
base_directory = argv[2]
if not os.path.exists(base_directory):
sys.exit("*** bad <base_directory>: %s" % base_directory)
if len(argv) == 3:
verbose = False
elif argv[3] == 'verbose':
verbose = True
else:
sys.exit("*** unknown argument: %s" % argv[2])
exec_report(base_directory, verbose)
################################################################################
# query git for year of last change
################################################################################
GIT_LOG_CMD = "git log --pretty=format:%%ai %s"
def call_git_log(filename):
out = subprocess.check_output((GIT_LOG_CMD % filename).split(' '))
return out.decode("utf-8").split('\n')
def get_git_change_years(filename):
git_log_lines = call_git_log(filename)
if len(git_log_lines) == 0:
return [datetime.date.today().year]
# timestamp is in ISO 8601 format. e.g. "2016-09-05 14:25:32 -0600"
return [line.split(' ')[0].split('-')[0] for line in git_log_lines]
def get_most_recent_git_change_year(filename):
return max(get_git_change_years(filename))
################################################################################
# read and write to file
################################################################################
def read_file_lines(filename):
f = open(filename, 'r', encoding="utf8")
file_lines = f.readlines()
f.close()
return file_lines
def write_file_lines(filename, file_lines):
f = open(filename, 'w', encoding="utf8")
f.write(''.join(file_lines))
f.close()
################################################################################
# update header years execution
################################################################################
COPYRIGHT = r'Copyright \(c\)'
YEAR = "20[0-9][0-9]"
YEAR_RANGE = '(%s)(-%s)?' % (YEAR, YEAR)
HOLDER = 'The DigiByte Core developers'
UPDATEABLE_LINE_COMPILED = re.compile(' '.join([COPYRIGHT, YEAR_RANGE, HOLDER]))
def get_updatable_copyright_line(file_lines):
index = 0
for line in file_lines:
if UPDATEABLE_LINE_COMPILED.search(line) is not None:
return index, line
index = index + 1
return None, None
def parse_year_range(year_range):
year_split = year_range.split('-')
start_year = year_split[0]
if len(year_split) == 1:
return start_year, start_year
return start_year, year_split[1]
def year_range_to_str(start_year, end_year):
if start_year == end_year:
return start_year
return "%s-%s" % (start_year, end_year)
def create_updated_copyright_line(line, last_git_change_year):
copyright_splitter = 'Copyright (c) '
copyright_split = line.split(copyright_splitter)
# Preserve characters on line that are ahead of the start of the copyright
# notice - they are part of the comment block and vary from file-to-file.
before_copyright = copyright_split[0]
after_copyright = copyright_split[1]
space_split = after_copyright.split(' ')
year_range = space_split[0]
start_year, end_year = parse_year_range(year_range)
if end_year == last_git_change_year:
return line
return (before_copyright + copyright_splitter +
year_range_to_str(start_year, last_git_change_year) + ' ' +
' '.join(space_split[1:]))
def update_updatable_copyright(filename):
file_lines = read_file_lines(filename)
index, line = get_updatable_copyright_line(file_lines)
if not line:
print_file_action_message(filename, "No updatable copyright.")
return
last_git_change_year = get_most_recent_git_change_year(filename)
new_line = create_updated_copyright_line(line, last_git_change_year)
if line == new_line:
print_file_action_message(filename, "Copyright up-to-date.")
return
file_lines[index] = new_line
write_file_lines(filename, file_lines)
print_file_action_message(filename,
"Copyright updated! -> %s" % last_git_change_year)
def exec_update_header_year(base_directory):
for filename in get_filenames_to_examine(base_directory):
update_updatable_copyright(filename)
################################################################################
# update cmd
################################################################################
UPDATE_USAGE = """
Updates all the copyright headers of "The DigiByte Core developers" which were
changed in a year more recent than is listed. For example:
// Copyright (c) <firstYear>-<lastYear> The DigiByte Core developers
will be updated to:
// Copyright (c) <firstYear>-<lastModifiedYear> The DigiByte Core developers
where <lastModifiedYear> is obtained from the 'git log' history.
This subcommand also handles copyright headers that have only a single year. In those cases:
// Copyright (c) <year> The DigiByte Core developers
will be updated to:
// Copyright (c) <year>-<lastModifiedYear> The DigiByte Core developers
where the update is appropriate.
Usage:
$ ./copyright_header.py update <base_directory>
Arguments:
<base_directory> - The base directory of a digibyte source code repository.
"""
def print_file_action_message(filename, action):
print("%-52s %s" % (filename, action))
def update_cmd(argv):
if len(argv) != 3:
sys.exit(UPDATE_USAGE)
base_directory = argv[2]
if not os.path.exists(base_directory):
sys.exit("*** bad base_directory: %s" % base_directory)
exec_update_header_year(base_directory)
################################################################################
# inserted copyright header format
################################################################################
def get_header_lines(header, start_year, end_year):
lines = header.split('\n')[1:-1]
lines[0] = lines[0] % year_range_to_str(start_year, end_year)
return [line + '\n' for line in lines]
CPP_HEADER = '''
// Copyright (c) %s The DigiByte Core developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
def get_cpp_header_lines_to_insert(start_year, end_year):
return reversed(get_header_lines(CPP_HEADER, start_year, end_year))
SCRIPT_HEADER = '''
# Copyright (c) %s The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
def get_script_header_lines_to_insert(start_year, end_year):
return reversed(get_header_lines(SCRIPT_HEADER, start_year, end_year))
################################################################################
# query git for year of last change
################################################################################
def get_git_change_year_range(filename):
years = get_git_change_years(filename)
return min(years), max(years)
################################################################################
# check for existing core copyright
################################################################################
def file_already_has_core_copyright(file_lines):
index, _ = get_updatable_copyright_line(file_lines)
return index is not None
################################################################################
# insert header execution
################################################################################
def file_has_hashbang(file_lines):
if len(file_lines) < 1:
return False
if len(file_lines[0]) <= 2:
return False
return file_lines[0][:2] == '#!'
def insert_script_header(filename, file_lines, start_year, end_year):
if file_has_hashbang(file_lines):
insert_idx = 1
else:
insert_idx = 0
header_lines = get_script_header_lines_to_insert(start_year, end_year)
for line in header_lines:
file_lines.insert(insert_idx, line)
write_file_lines(filename, file_lines)
def insert_cpp_header(filename, file_lines, start_year, end_year):
file_lines.insert(0, '\n')
header_lines = get_cpp_header_lines_to_insert(start_year, end_year)
for line in header_lines:
file_lines.insert(0, line)
write_file_lines(filename, file_lines)
def exec_insert_header(filename, style):
file_lines = read_file_lines(filename)
if file_already_has_core_copyright(file_lines):
sys.exit('*** %s already has a copyright by The DigiByte Core developers'
% (filename))
start_year, end_year = get_git_change_year_range(filename)
if style in ['python', 'shell']:
insert_script_header(filename, file_lines, start_year, end_year)
else:
insert_cpp_header(filename, file_lines, start_year, end_year)
################################################################################
# insert cmd
################################################################################
INSERT_USAGE = """
Inserts a copyright header for "The DigiByte Core developers" at the top of the
file in either Python or C++ style as determined by the file extension. If the
file is a Python file and it has a '#!' starting the first line, the header is
inserted in the line below it.
The copyright dates will be set to be:
"<year_introduced>-<current_year>"
where <year_introduced> is according to the 'git log' history. If
<year_introduced> is equal to <current_year>, the date will be set to be:
"<current_year>"
If the file already has a copyright for "The DigiByte Core developers", the
script will exit.
Usage:
$ ./copyright_header.py insert <file>
Arguments:
<file> - A source file in the digibyte repository.
"""
def insert_cmd(argv):
if len(argv) != 3:
sys.exit(INSERT_USAGE)
filename = argv[2]
if not os.path.isfile(filename):
sys.exit("*** bad filename: %s" % filename)
_, extension = os.path.splitext(filename)
if extension not in ['.h', '.cpp', '.cc', '.c', '.py', '.sh']:
sys.exit("*** cannot insert for file extension %s" % extension)
if extension == '.py':
style = 'python'
elif extension == '.sh':
style = 'shell'
else:
style = 'cpp'
exec_insert_header(filename, style)
################################################################################
# UI
################################################################################
USAGE = """
copyright_header.py - utilities for managing copyright headers of 'The DigiByte
Core developers' in repository source files.
Usage:
$ ./copyright_header <subcommand>
Subcommands:
report
update
insert
To see subcommand usage, run them without arguments.
"""
SUBCOMMANDS = ['report', 'update', 'insert']
if __name__ == "__main__":
if len(sys.argv) == 1:
sys.exit(USAGE)
subcommand = sys.argv[1]
if subcommand not in SUBCOMMANDS:
sys.exit(USAGE)
if subcommand == 'report':
report_cmd(sys.argv)
elif subcommand == 'update':
update_cmd(sys.argv)
elif subcommand == 'insert':
insert_cmd(sys.argv)

53
contrib/devtools/gen-manpages.sh Executable file
View file

@ -0,0 +1,53 @@
#!/usr/bin/env bash
# Copyright (c) 2016-2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
TOPDIR=${TOPDIR:-$(git rev-parse --show-toplevel)}
BUILDDIR=${BUILDDIR:-$TOPDIR}
BINDIR=${BINDIR:-$BUILDDIR/src}
MANDIR=${MANDIR:-$TOPDIR/doc/man}
DIGIBYTED=${DIGIBYTED:-$BINDIR/digibyted}
DIGIBYTECLI=${DIGIBYTECLI:-$BINDIR/digibyte-cli}
DIGIBYTETX=${DIGIBYTETX:-$BINDIR/digibyte-tx}
WALLET_TOOL=${WALLET_TOOL:-$BINDIR/digibyte-wallet}
DIGIBYTEUTIL=${DIGIBYTEQT:-$BINDIR/digibyte-util}
DIGIBYTEQT=${DIGIBYTEQT:-$BINDIR/qt/digibyte-qt}
[ ! -x $DIGIBYTED ] && echo "$DIGIBYTED not found or not executable." && exit 1
# Don't allow man pages to be generated for binaries built from a dirty tree
DIRTY=""
for cmd in $DIGIBYTED $DIGIBYTECLI $DIGIBYTETX $WALLET_TOOL $DIGIBYTEUTIL $DIGIBYTEQT; do
VERSION_OUTPUT=$($cmd --version)
if [[ $VERSION_OUTPUT == *"dirty"* ]]; then
DIRTY="${DIRTY}${cmd}\n"
fi
done
if [ -n "$DIRTY" ]
then
echo -e "WARNING: the following binaries were built from a dirty tree:\n"
echo -e $DIRTY
echo "man pages generated from dirty binaries should NOT be committed."
echo "To properly generate man pages, please commit your changes to the above binaries, rebuild them, then run this script again."
fi
# The autodetected version git tag can screw up manpage output a little bit
read -r -a DGBVER <<< "$($DIGIBYTECLI --version | head -n1 | awk -F'[ -]' '{ print $6, $7 }')"
# Create a footer file with copyright content.
# This gets autodetected fine for digibyted if --version-string is not set,
# but has different outcomes for digibyte-qt and digibyte-cli.
echo "[COPYRIGHT]" > footer.h2m
$DIGIBYTED --version | sed -n '1!p' >> footer.h2m
for cmd in $DIGIBYTED $DIGIBYTECLI $DIGIBYTETX $WALLET_TOOL $DIGIBYTEUTIL $DIGIBYTEQT; do
cmdname="${cmd##*/}"
help2man -N --version-string=${DGBVER[0]} --include=footer.h2m -o ${MANDIR}/${cmdname}.1 ${cmd}
sed -i "s/\\\-${DGBVER[1]}//g" ${MANDIR}/${cmdname}.1
done
rm -f footer.h2m

331
contrib/devtools/github-merge.py Executable file
View file

@ -0,0 +1,331 @@
#!/usr/bin/env python3
# Copyright (c) 2016-2017 DigiByte Core Developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
# This script will locally construct a merge commit for a pull request on a
# github repository, inspect it, sign it and optionally push it.
# The following temporary branches are created/overwritten and deleted:
# * pull/$PULL/base (the current master we're merging onto)
# * pull/$PULL/head (the current state of the remote pull request)
# * pull/$PULL/merge (github's merge)
# * pull/$PULL/local-merge (our merge)
# In case of a clean merge that is accepted by the user, the local branch with
# name $BRANCH is overwritten with the merged result, and optionally pushed.
from __future__ import division,print_function,unicode_literals
import os
from sys import stdin,stdout,stderr
import argparse
import hashlib
import subprocess
import sys
import json
import codecs
try:
from urllib.request import Request,urlopen
except:
from urllib2 import Request,urlopen
# External tools (can be overridden using environment)
GIT = os.getenv('GIT','git')
BASH = os.getenv('BASH','bash')
# OS specific configuration for terminal attributes
ATTR_RESET = ''
ATTR_PR = ''
COMMIT_FORMAT = '%h %s (%an)%d'
if os.name == 'posix': # if posix, assume we can use basic terminal escapes
ATTR_RESET = '\033[0m'
ATTR_PR = '\033[1;36m'
COMMIT_FORMAT = '%C(bold blue)%h%Creset %s %C(cyan)(%an)%Creset%C(green)%d%Creset'
def git_config_get(option, default=None):
'''
Get named configuration option from git repository.
'''
try:
return subprocess.check_output([GIT,'config','--get',option]).rstrip().decode('utf-8')
except subprocess.CalledProcessError:
return default
def retrieve_pr_info(repo,pull):
'''
Retrieve pull request information from github.
Return None if no title can be found, or an error happens.
'''
try:
req = Request("https://api.github.com/repos/"+repo+"/pulls/"+pull)
result = urlopen(req)
reader = codecs.getreader('utf-8')
obj = json.load(reader(result))
return obj
except Exception as e:
print('Warning: unable to retrieve pull information from github: %s' % e)
return None
def ask_prompt(text):
print(text,end=" ",file=stderr)
stderr.flush()
reply = stdin.readline().rstrip()
print("",file=stderr)
return reply
def get_symlink_files():
files = sorted(subprocess.check_output([GIT, 'ls-tree', '--full-tree', '-r', 'HEAD']).splitlines())
ret = []
for f in files:
if (int(f.decode('utf-8').split(" ")[0], 8) & 0o170000) == 0o120000:
ret.append(f.decode('utf-8').split("\t")[1])
return ret
def tree_sha512sum(commit='HEAD'):
# request metadata for entire tree, recursively
files = []
blob_by_name = {}
for line in subprocess.check_output([GIT, 'ls-tree', '--full-tree', '-r', commit]).splitlines():
name_sep = line.index(b'\t')
metadata = line[:name_sep].split() # perms, 'blob', blobid
assert(metadata[1] == b'blob')
name = line[name_sep+1:]
files.append(name)
blob_by_name[name] = metadata[2]
files.sort()
# open connection to git-cat-file in batch mode to request data for all blobs
# this is much faster than launching it per file
p = subprocess.Popen([GIT, 'cat-file', '--batch'], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
overall = hashlib.sha512()
for f in files:
blob = blob_by_name[f]
# request blob
p.stdin.write(blob + b'\n')
p.stdin.flush()
# read header: blob, "blob", size
reply = p.stdout.readline().split()
assert(reply[0] == blob and reply[1] == b'blob')
size = int(reply[2])
# hash the blob data
intern = hashlib.sha512()
ptr = 0
while ptr < size:
bs = min(65536, size - ptr)
piece = p.stdout.read(bs)
if len(piece) == bs:
intern.update(piece)
else:
raise IOError('Premature EOF reading git cat-file output')
ptr += bs
dig = intern.hexdigest()
assert(p.stdout.read(1) == b'\n') # ignore LF that follows blob data
# update overall hash with file hash
overall.update(dig.encode("utf-8"))
overall.update(" ".encode("utf-8"))
overall.update(f)
overall.update("\n".encode("utf-8"))
p.stdin.close()
if p.wait():
raise IOError('Non-zero return value executing git cat-file')
return overall.hexdigest()
def print_merge_details(pull, title, branch, base_branch, head_branch):
print('%s#%s%s %s %sinto %s%s' % (ATTR_RESET+ATTR_PR,pull,ATTR_RESET,title,ATTR_RESET+ATTR_PR,branch,ATTR_RESET))
subprocess.check_call([GIT,'log','--graph','--topo-order','--pretty=format:'+COMMIT_FORMAT,base_branch+'..'+head_branch])
def parse_arguments():
epilog = '''
In addition, you can set the following git configuration variables:
githubmerge.repository (mandatory),
user.signingkey (mandatory),
githubmerge.host (default: git@github.com),
githubmerge.branch (no default),
githubmerge.testcmd (default: none).
'''
parser = argparse.ArgumentParser(description='Utility to merge, sign and push github pull requests',
epilog=epilog)
parser.add_argument('pull', metavar='PULL', type=int, nargs=1,
help='Pull request ID to merge')
parser.add_argument('branch', metavar='BRANCH', type=str, nargs='?',
default=None, help='Branch to merge against (default: githubmerge.branch setting, or base branch for pull, or \'master\')')
return parser.parse_args()
def main():
# Extract settings from git repo
repo = git_config_get('githubmerge.repository')
host = git_config_get('githubmerge.host','git@github.com')
opt_branch = git_config_get('githubmerge.branch',None)
testcmd = git_config_get('githubmerge.testcmd')
signingkey = git_config_get('user.signingkey')
if repo is None:
print("ERROR: No repository configured. Use this command to set:", file=stderr)
print("git config githubmerge.repository <owner>/<repo>", file=stderr)
sys.exit(1)
if signingkey is None:
print("ERROR: No GPG signing key set. Set one using:",file=stderr)
print("git config --global user.signingkey <key>",file=stderr)
sys.exit(1)
host_repo = host+":"+repo # shortcut for push/pull target
# Extract settings from command line
args = parse_arguments()
pull = str(args.pull[0])
# Receive pull information from github
info = retrieve_pr_info(repo,pull)
if info is None:
sys.exit(1)
title = info['title'].strip()
body = info['body'].strip()
# precedence order for destination branch argument:
# - command line argument
# - githubmerge.branch setting
# - base branch for pull (as retrieved from github)
# - 'master'
branch = args.branch or opt_branch or info['base']['ref'] or 'master'
# Initialize source branches
head_branch = 'pull/'+pull+'/head'
base_branch = 'pull/'+pull+'/base'
merge_branch = 'pull/'+pull+'/merge'
local_merge_branch = 'pull/'+pull+'/local-merge'
devnull = open(os.devnull, 'w', encoding="utf8")
try:
subprocess.check_call([GIT,'checkout','-q',branch])
except subprocess.CalledProcessError:
print("ERROR: Cannot check out branch %s." % (branch), file=stderr)
sys.exit(3)
try:
subprocess.check_call([GIT,'fetch','-q',host_repo,'+refs/pull/'+pull+'/*:refs/heads/pull/'+pull+'/*',
'+refs/heads/'+branch+':refs/heads/'+base_branch])
except subprocess.CalledProcessError:
print("ERROR: Cannot find pull request #%s or branch %s on %s." % (pull,branch,host_repo), file=stderr)
sys.exit(3)
try:
subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+head_branch], stdout=devnull, stderr=stdout)
except subprocess.CalledProcessError:
print("ERROR: Cannot find head of pull request #%s on %s." % (pull,host_repo), file=stderr)
sys.exit(3)
try:
subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+merge_branch], stdout=devnull, stderr=stdout)
except subprocess.CalledProcessError:
print("ERROR: Cannot find merge of pull request #%s on %s." % (pull,host_repo), file=stderr)
sys.exit(3)
subprocess.check_call([GIT,'checkout','-q',base_branch])
subprocess.call([GIT,'branch','-q','-D',local_merge_branch], stderr=devnull)
subprocess.check_call([GIT,'checkout','-q','-b',local_merge_branch])
try:
# Go up to the repository's root.
toplevel = subprocess.check_output([GIT,'rev-parse','--show-toplevel']).strip()
os.chdir(toplevel)
# Create unsigned merge commit.
if title:
firstline = 'Merge #%s: %s' % (pull,title)
else:
firstline = 'Merge #%s' % (pull,)
message = firstline + '\n\n'
message += subprocess.check_output([GIT,'log','--no-merges','--topo-order','--pretty=format:%h %s (%an)',base_branch+'..'+head_branch]).decode('utf-8')
message += '\n\nPull request description:\n\n ' + body.replace('\n', '\n ') + '\n'
try:
subprocess.check_call([GIT,'merge','-q','--commit','--no-edit','--no-ff','-m',message.encode('utf-8'),head_branch])
except subprocess.CalledProcessError:
print("ERROR: Cannot be merged cleanly.",file=stderr)
subprocess.check_call([GIT,'merge','--abort'])
sys.exit(4)
logmsg = subprocess.check_output([GIT,'log','--pretty=format:%s','-n','1']).decode('utf-8')
if logmsg.rstrip() != firstline.rstrip():
print("ERROR: Creating merge failed (already merged?).",file=stderr)
sys.exit(4)
symlink_files = get_symlink_files()
for f in symlink_files:
print("ERROR: File %s was a symlink" % f)
if len(symlink_files) > 0:
sys.exit(4)
# Put tree SHA512 into the message
try:
first_sha512 = tree_sha512sum()
message += '\n\nTree-SHA512: ' + first_sha512
except subprocess.CalledProcessError:
print("ERROR: Unable to compute tree hash")
sys.exit(4)
try:
subprocess.check_call([GIT,'commit','--amend','-m',message.encode('utf-8')])
except subprocess.CalledProcessError:
print("ERROR: Cannot update message.", file=stderr)
sys.exit(4)
print_merge_details(pull, title, branch, base_branch, head_branch)
print()
# Run test command if configured.
if testcmd:
if subprocess.call(testcmd,shell=True):
print("ERROR: Running %s failed." % testcmd,file=stderr)
sys.exit(5)
# Show the created merge.
diff = subprocess.check_output([GIT,'diff',merge_branch+'..'+local_merge_branch])
subprocess.check_call([GIT,'diff',base_branch+'..'+local_merge_branch])
if diff:
print("WARNING: merge differs from github!",file=stderr)
reply = ask_prompt("Type 'ignore' to continue.")
if reply.lower() == 'ignore':
print("Difference with github ignored.",file=stderr)
else:
sys.exit(6)
else:
# Verify the result manually.
print("Dropping you on a shell so you can try building/testing the merged source.",file=stderr)
print("Run 'git diff HEAD~' to show the changes being merged.",file=stderr)
print("Type 'exit' when done.",file=stderr)
if os.path.isfile('/etc/debian_version'): # Show pull number on Debian default prompt
os.putenv('debian_chroot',pull)
subprocess.call([BASH,'-i'])
second_sha512 = tree_sha512sum()
if first_sha512 != second_sha512:
print("ERROR: Tree hash changed unexpectedly",file=stderr)
sys.exit(8)
# Sign the merge commit.
print_merge_details(pull, title, branch, base_branch, head_branch)
while True:
reply = ask_prompt("Type 's' to sign off on the above merge, or 'x' to reject and exit.").lower()
if reply == 's':
try:
subprocess.check_call([GIT,'commit','-q','--gpg-sign','--amend','--no-edit'])
break
except subprocess.CalledProcessError:
print("Error while signing, asking again.",file=stderr)
elif reply == 'x':
print("Not signing off on merge, exiting.",file=stderr)
sys.exit(1)
# Put the result in branch.
subprocess.check_call([GIT,'checkout','-q',branch])
subprocess.check_call([GIT,'reset','-q','--hard',local_merge_branch])
finally:
# Clean up temporary branches.
subprocess.call([GIT,'checkout','-q',branch])
subprocess.call([GIT,'branch','-q','-D',head_branch],stderr=devnull)
subprocess.call([GIT,'branch','-q','-D',base_branch],stderr=devnull)
subprocess.call([GIT,'branch','-q','-D',merge_branch],stderr=devnull)
subprocess.call([GIT,'branch','-q','-D',local_merge_branch],stderr=devnull)
# Push the result.
while True:
reply = ask_prompt("Type 'push' to push the result to %s, branch %s, or 'x' to exit without pushing." % (host_repo,branch)).lower()
if reply == 'push':
subprocess.check_call([GIT,'push',host_repo,'refs/heads/'+branch])
break
elif reply == 'x':
sys.exit(1)
if __name__ == '__main__':
main()

76
contrib/devtools/optimize-pngs.py Executable file
View file

@ -0,0 +1,76 @@
#!/usr/bin/env python3
# Copyright (c) 2014-2018 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Run this script every time you change one of the png files. Using pngcrush, it will optimize the png files, remove various color profiles, remove ancillary chunks (alla) and text chunks (text).
#pngcrush -brute -ow -rem gAMA -rem cHRM -rem iCCP -rem sRGB -rem alla -rem text
'''
import os
import sys
import subprocess
import hashlib
from PIL import Image # pip3 install Pillow
def file_hash(filename):
'''Return hash of raw file contents'''
with open(filename, 'rb') as f:
return hashlib.sha256(f.read()).hexdigest()
def content_hash(filename):
'''Return hash of RGBA contents of image'''
i = Image.open(filename)
i = i.convert('RGBA')
data = i.tobytes()
return hashlib.sha256(data).hexdigest()
pngcrush = 'pngcrush'
git = 'git'
folders = ["src/qt/res/movies", "src/qt/res/icons", "share/pixmaps"]
basePath = subprocess.check_output([git, 'rev-parse', '--show-toplevel'], universal_newlines=True).rstrip('\n')
totalSaveBytes = 0
noHashChange = True
outputArray = []
for folder in folders:
absFolder=os.path.join(basePath, folder)
for file in os.listdir(absFolder):
extension = os.path.splitext(file)[1]
if extension.lower() == '.png':
print("optimizing {}...".format(file), end =' ')
file_path = os.path.join(absFolder, file)
fileMetaMap = {'file' : file, 'osize': os.path.getsize(file_path), 'sha256Old' : file_hash(file_path)}
fileMetaMap['contentHashPre'] = content_hash(file_path)
try:
subprocess.call([pngcrush, "-brute", "-ow", "-rem", "gAMA", "-rem", "cHRM", "-rem", "iCCP", "-rem", "sRGB", "-rem", "alla", "-rem", "text", file_path],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
except:
print("pngcrush is not installed, aborting...")
sys.exit(0)
#verify
if "Not a PNG file" in subprocess.check_output([pngcrush, "-n", "-v", file_path], stderr=subprocess.STDOUT, universal_newlines=True):
print("PNG file "+file+" is corrupted after crushing, check out pngcursh version")
sys.exit(1)
fileMetaMap['sha256New'] = file_hash(file_path)
fileMetaMap['contentHashPost'] = content_hash(file_path)
if fileMetaMap['contentHashPre'] != fileMetaMap['contentHashPost']:
print("Image contents of PNG file {} before and after crushing don't match".format(file))
sys.exit(1)
fileMetaMap['psize'] = os.path.getsize(file_path)
outputArray.append(fileMetaMap)
print("done")
print("summary:\n+++++++++++++++++")
for fileDict in outputArray:
oldHash = fileDict['sha256Old']
newHash = fileDict['sha256New']
totalSaveBytes += fileDict['osize'] - fileDict['psize']
noHashChange = noHashChange and (oldHash == newHash)
print(fileDict['file']+"\n size diff from: "+str(fileDict['osize'])+" to: "+str(fileDict['psize'])+"\n old sha256: "+oldHash+"\n new sha256: "+newHash+"\n")
print("completed. Checksum stable: "+str(noHashChange)+". Total reduction: "+str(totalSaveBytes)+" bytes")

323
contrib/devtools/pixie.py Normal file
View file

@ -0,0 +1,323 @@
#!/usr/bin/env python3
# Copyright (c) 2020 Wladimir J. van der Laan
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Compact, self-contained ELF implementation for digibyte-core security checks.
'''
import struct
import types
from typing import Dict, List, Optional, Union, Tuple
# you can find all these values in elf.h
EI_NIDENT = 16
# Byte indices in e_ident
EI_CLASS = 4 # ELFCLASSxx
EI_DATA = 5 # ELFDATAxxxx
ELFCLASS32 = 1 # 32-bit
ELFCLASS64 = 2 # 64-bit
ELFDATA2LSB = 1 # little endian
ELFDATA2MSB = 2 # big endian
# relevant values for e_machine
EM_386 = 3
EM_PPC64 = 21
EM_ARM = 40
EM_AARCH64 = 183
EM_X86_64 = 62
EM_RISCV = 243
# relevant values for e_type
ET_DYN = 3
# relevant values for sh_type
SHT_PROGBITS = 1
SHT_STRTAB = 3
SHT_DYNAMIC = 6
SHT_DYNSYM = 11
SHT_GNU_verneed = 0x6ffffffe
SHT_GNU_versym = 0x6fffffff
# relevant values for p_type
PT_LOAD = 1
PT_GNU_STACK = 0x6474e551
PT_GNU_RELRO = 0x6474e552
# relevant values for p_flags
PF_X = (1 << 0)
PF_W = (1 << 1)
PF_R = (1 << 2)
# relevant values for d_tag
DT_NEEDED = 1
DT_FLAGS = 30
# relevant values of `d_un.d_val' in the DT_FLAGS entry
DF_BIND_NOW = 0x00000008
# relevant d_tags with string payload
STRING_TAGS = {DT_NEEDED}
# rrlevant values for ST_BIND subfield of st_info (symbol binding)
STB_LOCAL = 0
class ELFRecord(types.SimpleNamespace):
'''Unified parsing for ELF records.'''
def __init__(self, data: bytes, offset: int, eh: 'ELFHeader', total_size: Optional[int]) -> None:
hdr_struct = self.STRUCT[eh.ei_class][0][eh.ei_data]
if total_size is not None and hdr_struct.size > total_size:
raise ValueError(f'{self.__class__.__name__} header size too small ({total_size} < {hdr_struct.size})')
for field, value in zip(self.STRUCT[eh.ei_class][1], hdr_struct.unpack(data[offset:offset + hdr_struct.size])):
setattr(self, field, value)
def BiStruct(chars: str) -> Dict[int, struct.Struct]:
'''Compile a struct parser for both endians.'''
return {
ELFDATA2LSB: struct.Struct('<' + chars),
ELFDATA2MSB: struct.Struct('>' + chars),
}
class ELFHeader(ELFRecord):
FIELDS = ['e_type', 'e_machine', 'e_version', 'e_entry', 'e_phoff', 'e_shoff', 'e_flags', 'e_ehsize', 'e_phentsize', 'e_phnum', 'e_shentsize', 'e_shnum', 'e_shstrndx']
STRUCT = {
ELFCLASS32: (BiStruct('HHIIIIIHHHHHH'), FIELDS),
ELFCLASS64: (BiStruct('HHIQQQIHHHHHH'), FIELDS),
}
def __init__(self, data: bytes, offset: int) -> None:
self.e_ident = data[offset:offset + EI_NIDENT]
if self.e_ident[0:4] != b'\x7fELF':
raise ValueError('invalid ELF magic')
self.ei_class = self.e_ident[EI_CLASS]
self.ei_data = self.e_ident[EI_DATA]
super().__init__(data, offset + EI_NIDENT, self, None)
def __repr__(self) -> str:
return f'Header(e_ident={self.e_ident!r}, e_type={self.e_type}, e_machine={self.e_machine}, e_version={self.e_version}, e_entry={self.e_entry}, e_phoff={self.e_phoff}, e_shoff={self.e_shoff}, e_flags={self.e_flags}, e_ehsize={self.e_ehsize}, e_phentsize={self.e_phentsize}, e_phnum={self.e_phnum}, e_shentsize={self.e_shentsize}, e_shnum={self.e_shnum}, e_shstrndx={self.e_shstrndx})'
class Section(ELFRecord):
name: Optional[bytes] = None
FIELDS = ['sh_name', 'sh_type', 'sh_flags', 'sh_addr', 'sh_offset', 'sh_size', 'sh_link', 'sh_info', 'sh_addralign', 'sh_entsize']
STRUCT = {
ELFCLASS32: (BiStruct('IIIIIIIIII'), FIELDS),
ELFCLASS64: (BiStruct('IIQQQQIIQQ'), FIELDS),
}
def __init__(self, data: bytes, offset: int, eh: ELFHeader) -> None:
super().__init__(data, offset, eh, eh.e_shentsize)
self._data = data
def __repr__(self) -> str:
return f'Section(sh_name={self.sh_name}({self.name!r}), sh_type=0x{self.sh_type:x}, sh_flags={self.sh_flags}, sh_addr=0x{self.sh_addr:x}, sh_offset=0x{self.sh_offset:x}, sh_size={self.sh_size}, sh_link={self.sh_link}, sh_info={self.sh_info}, sh_addralign={self.sh_addralign}, sh_entsize={self.sh_entsize})'
def contents(self) -> bytes:
'''Return section contents.'''
return self._data[self.sh_offset:self.sh_offset + self.sh_size]
class ProgramHeader(ELFRecord):
STRUCT = {
# different ELF classes have the same fields, but in a different order to optimize space versus alignment
ELFCLASS32: (BiStruct('IIIIIIII'), ['p_type', 'p_offset', 'p_vaddr', 'p_paddr', 'p_filesz', 'p_memsz', 'p_flags', 'p_align']),
ELFCLASS64: (BiStruct('IIQQQQQQ'), ['p_type', 'p_flags', 'p_offset', 'p_vaddr', 'p_paddr', 'p_filesz', 'p_memsz', 'p_align']),
}
def __init__(self, data: bytes, offset: int, eh: ELFHeader) -> None:
super().__init__(data, offset, eh, eh.e_phentsize)
def __repr__(self) -> str:
return f'ProgramHeader(p_type={self.p_type}, p_offset={self.p_offset}, p_vaddr={self.p_vaddr}, p_paddr={self.p_paddr}, p_filesz={self.p_filesz}, p_memsz={self.p_memsz}, p_flags={self.p_flags}, p_align={self.p_align})'
class Symbol(ELFRecord):
STRUCT = {
# different ELF classes have the same fields, but in a different order to optimize space versus alignment
ELFCLASS32: (BiStruct('IIIBBH'), ['st_name', 'st_value', 'st_size', 'st_info', 'st_other', 'st_shndx']),
ELFCLASS64: (BiStruct('IBBHQQ'), ['st_name', 'st_info', 'st_other', 'st_shndx', 'st_value', 'st_size']),
}
def __init__(self, data: bytes, offset: int, eh: ELFHeader, symtab: Section, strings: bytes, version: Optional[bytes]) -> None:
super().__init__(data, offset, eh, symtab.sh_entsize)
self.name = _lookup_string(strings, self.st_name)
self.version = version
def __repr__(self) -> str:
return f'Symbol(st_name={self.st_name}({self.name!r}), st_value={self.st_value}, st_size={self.st_size}, st_info={self.st_info}, st_other={self.st_other}, st_shndx={self.st_shndx}, version={self.version!r})'
@property
def is_import(self) -> bool:
'''Returns whether the symbol is an imported symbol.'''
return self.st_bind != STB_LOCAL and self.st_shndx == 0
@property
def is_export(self) -> bool:
'''Returns whether the symbol is an exported symbol.'''
return self.st_bind != STB_LOCAL and self.st_shndx != 0
@property
def st_bind(self) -> int:
'''Returns STB_*.'''
return self.st_info >> 4
class Verneed(ELFRecord):
DEF = (BiStruct('HHIII'), ['vn_version', 'vn_cnt', 'vn_file', 'vn_aux', 'vn_next'])
STRUCT = { ELFCLASS32: DEF, ELFCLASS64: DEF }
def __init__(self, data: bytes, offset: int, eh: ELFHeader) -> None:
super().__init__(data, offset, eh, None)
def __repr__(self) -> str:
return f'Verneed(vn_version={self.vn_version}, vn_cnt={self.vn_cnt}, vn_file={self.vn_file}, vn_aux={self.vn_aux}, vn_next={self.vn_next})'
class Vernaux(ELFRecord):
DEF = (BiStruct('IHHII'), ['vna_hash', 'vna_flags', 'vna_other', 'vna_name', 'vna_next'])
STRUCT = { ELFCLASS32: DEF, ELFCLASS64: DEF }
def __init__(self, data: bytes, offset: int, eh: ELFHeader, strings: bytes) -> None:
super().__init__(data, offset, eh, None)
self.name = _lookup_string(strings, self.vna_name)
def __repr__(self) -> str:
return f'Veraux(vna_hash={self.vna_hash}, vna_flags={self.vna_flags}, vna_other={self.vna_other}, vna_name={self.vna_name}({self.name!r}), vna_next={self.vna_next})'
class DynTag(ELFRecord):
STRUCT = {
ELFCLASS32: (BiStruct('II'), ['d_tag', 'd_val']),
ELFCLASS64: (BiStruct('QQ'), ['d_tag', 'd_val']),
}
def __init__(self, data: bytes, offset: int, eh: ELFHeader, section: Section) -> None:
super().__init__(data, offset, eh, section.sh_entsize)
def __repr__(self) -> str:
return f'DynTag(d_tag={self.d_tag}, d_val={self.d_val})'
def _lookup_string(data: bytes, index: int) -> bytes:
'''Look up string by offset in ELF string table.'''
endx = data.find(b'\x00', index)
assert endx != -1
return data[index:endx]
VERSYM_S = BiStruct('H') # .gnu_version section has a single 16-bit integer per symbol in the linked section
def _parse_symbol_table(section: Section, strings: bytes, eh: ELFHeader, versym: bytes, verneed: Dict[int, bytes]) -> List[Symbol]:
'''Parse symbol table, return a list of symbols.'''
data = section.contents()
symbols = []
versym_iter = (verneed.get(v[0]) for v in VERSYM_S[eh.ei_data].iter_unpack(versym))
for ofs, version in zip(range(0, len(data), section.sh_entsize), versym_iter):
symbols.append(Symbol(data, ofs, eh, section, strings, version))
return symbols
def _parse_verneed(section: Section, strings: bytes, eh: ELFHeader) -> Dict[int, bytes]:
'''Parse .gnu.version_r section, return a dictionary of {versym: 'GLIBC_...'}.'''
data = section.contents()
ofs = 0
result = {}
while True:
verneed = Verneed(data, ofs, eh)
aofs = ofs + verneed.vn_aux
while True:
vernaux = Vernaux(data, aofs, eh, strings)
result[vernaux.vna_other] = vernaux.name
if not vernaux.vna_next:
break
aofs += vernaux.vna_next
if not verneed.vn_next:
break
ofs += verneed.vn_next
return result
def _parse_dyn_tags(section: Section, strings: bytes, eh: ELFHeader) -> List[Tuple[int, Union[bytes, int]]]:
'''Parse dynamic tags. Return array of tuples.'''
data = section.contents()
ofs = 0
result = []
for ofs in range(0, len(data), section.sh_entsize):
tag = DynTag(data, ofs, eh, section)
val = _lookup_string(strings, tag.d_val) if tag.d_tag in STRING_TAGS else tag.d_val
result.append((tag.d_tag, val))
return result
class ELFFile:
sections: List[Section]
program_headers: List[ProgramHeader]
dyn_symbols: List[Symbol]
dyn_tags: List[Tuple[int, Union[bytes, int]]]
def __init__(self, data: bytes) -> None:
self.data = data
self.hdr = ELFHeader(self.data, 0)
self._load_sections()
self._load_program_headers()
self._load_dyn_symbols()
self._load_dyn_tags()
self._section_to_segment_mapping()
def _load_sections(self) -> None:
self.sections = []
for idx in range(self.hdr.e_shnum):
offset = self.hdr.e_shoff + idx * self.hdr.e_shentsize
self.sections.append(Section(self.data, offset, self.hdr))
shstr = self.sections[self.hdr.e_shstrndx].contents()
for section in self.sections:
section.name = _lookup_string(shstr, section.sh_name)
def _load_program_headers(self) -> None:
self.program_headers = []
for idx in range(self.hdr.e_phnum):
offset = self.hdr.e_phoff + idx * self.hdr.e_phentsize
self.program_headers.append(ProgramHeader(self.data, offset, self.hdr))
def _load_dyn_symbols(self) -> None:
# first, load 'verneed' section
verneed = None
for section in self.sections:
if section.sh_type == SHT_GNU_verneed:
strtab = self.sections[section.sh_link].contents() # associated string table
assert verneed is None # only one section of this kind please
verneed = _parse_verneed(section, strtab, self.hdr)
assert verneed is not None
# then, correlate GNU versym sections with dynamic symbol sections
versym = {}
for section in self.sections:
if section.sh_type == SHT_GNU_versym:
versym[section.sh_link] = section
# finally, load dynsym sections
self.dyn_symbols = []
for idx, section in enumerate(self.sections):
if section.sh_type == SHT_DYNSYM: # find dynamic symbol tables
strtab_data = self.sections[section.sh_link].contents() # associated string table
versym_data = versym[idx].contents() # associated symbol version table
self.dyn_symbols += _parse_symbol_table(section, strtab_data, self.hdr, versym_data, verneed)
def _load_dyn_tags(self) -> None:
self.dyn_tags = []
for idx, section in enumerate(self.sections):
if section.sh_type == SHT_DYNAMIC: # find dynamic tag tables
strtab = self.sections[section.sh_link].contents() # associated string table
self.dyn_tags += _parse_dyn_tags(section, strtab, self.hdr)
def _section_to_segment_mapping(self) -> None:
for ph in self.program_headers:
ph.sections = []
for section in self.sections:
if ph.p_vaddr <= section.sh_addr < (ph.p_vaddr + ph.p_memsz):
ph.sections.append(section)
def query_dyn_tags(self, tag_in: int) -> List[Union[int, bytes]]:
'''Return the values of all dyn tags with the specified tag.'''
return [val for (tag, val) in self.dyn_tags if tag == tag_in]
def load(filename: str) -> ELFFile:
with open(filename, 'rb') as f:
data = f.read()
return ELFFile(data)

261
contrib/devtools/security-check.py Executable file
View file

@ -0,0 +1,261 @@
#!/usr/bin/env python3
# Copyright (c) 2015-2021 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Perform basic security checks on a series of executables.
Exit status will be 0 if successful, and the program will be silent.
Otherwise the exit status will be 1 and it will log which executables failed which checks.
'''
import sys
from typing import List, Optional
import lief
import pixie
def check_ELF_PIE(executable) -> bool:
'''
Check for position independent executable (PIE), allowing for address space randomization.
'''
elf = pixie.load(executable)
return elf.hdr.e_type == pixie.ET_DYN
def check_ELF_NX(executable) -> bool:
'''
Check that no sections are writable and executable (including the stack)
'''
elf = pixie.load(executable)
have_wx = False
have_gnu_stack = False
for ph in elf.program_headers:
if ph.p_type == pixie.PT_GNU_STACK:
have_gnu_stack = True
if (ph.p_flags & pixie.PF_W) != 0 and (ph.p_flags & pixie.PF_X) != 0: # section is both writable and executable
have_wx = True
return have_gnu_stack and not have_wx
def check_ELF_RELRO(executable) -> bool:
'''
Check for read-only relocations.
GNU_RELRO program header must exist
Dynamic section must have BIND_NOW flag
'''
elf = pixie.load(executable)
have_gnu_relro = False
for ph in elf.program_headers:
# Note: not checking p_flags == PF_R: here as linkers set the permission differently
# This does not affect security: the permission flags of the GNU_RELRO program
# header are ignored, the PT_LOAD header determines the effective permissions.
# However, the dynamic linker need to write to this area so these are RW.
# Glibc itself takes care of mprotecting this area R after relocations are finished.
# See also https://marc.info/?l=binutils&m=1498883354122353
if ph.p_type == pixie.PT_GNU_RELRO:
have_gnu_relro = True
have_bindnow = False
for flags in elf.query_dyn_tags(pixie.DT_FLAGS):
assert isinstance(flags, int)
if flags & pixie.DF_BIND_NOW:
have_bindnow = True
return have_gnu_relro and have_bindnow
def check_ELF_Canary(executable) -> bool:
'''
Check for use of stack canary
'''
elf = pixie.load(executable)
ok = False
for symbol in elf.dyn_symbols:
if symbol.name == b'__stack_chk_fail':
ok = True
return ok
def check_ELF_separate_code(executable):
'''
Check that sections are appropriately separated in virtual memory,
based on their permissions. This checks for missing -Wl,-z,separate-code
and potentially other problems.
'''
elf = pixie.load(executable)
R = pixie.PF_R
W = pixie.PF_W
E = pixie.PF_X
EXPECTED_FLAGS = {
# Read + execute
b'.init': R | E,
b'.plt': R | E,
b'.plt.got': R | E,
b'.plt.sec': R | E,
b'.text': R | E,
b'.fini': R | E,
# Read-only data
b'.interp': R,
b'.note.gnu.property': R,
b'.note.gnu.build-id': R,
b'.note.ABI-tag': R,
b'.gnu.hash': R,
b'.dynsym': R,
b'.dynstr': R,
b'.gnu.version': R,
b'.gnu.version_r': R,
b'.rela.dyn': R,
b'.rela.plt': R,
b'.rodata': R,
b'.eh_frame_hdr': R,
b'.eh_frame': R,
b'.qtmetadata': R,
b'.gcc_except_table': R,
b'.stapsdt.base': R,
# Writable data
b'.init_array': R | W,
b'.fini_array': R | W,
b'.dynamic': R | W,
b'.got': R | W,
b'.data': R | W,
b'.bss': R | W,
}
if elf.hdr.e_machine == pixie.EM_PPC64:
# .plt is RW on ppc64 even with separate-code
EXPECTED_FLAGS[b'.plt'] = R | W
# For all LOAD program headers get mapping to the list of sections,
# and for each section, remember the flags of the associated program header.
flags_per_section = {}
for ph in elf.program_headers:
if ph.p_type == pixie.PT_LOAD:
for section in ph.sections:
assert(section.name not in flags_per_section)
flags_per_section[section.name] = ph.p_flags
# Spot-check ELF LOAD program header flags per section
# If these sections exist, check them against the expected R/W/E flags
for (section, flags) in flags_per_section.items():
if section in EXPECTED_FLAGS:
if EXPECTED_FLAGS[section] != flags:
return False
return True
def check_PE_DYNAMIC_BASE(executable) -> bool:
'''PIE: DllCharacteristics bit 0x40 signifies dynamicbase (ASLR)'''
binary = lief.parse(executable)
return lief.PE.DLL_CHARACTERISTICS.DYNAMIC_BASE in binary.optional_header.dll_characteristics_lists
# Must support high-entropy 64-bit address space layout randomization
# in addition to DYNAMIC_BASE to have secure ASLR.
def check_PE_HIGH_ENTROPY_VA(executable) -> bool:
'''PIE: DllCharacteristics bit 0x20 signifies high-entropy ASLR'''
binary = lief.parse(executable)
return lief.PE.DLL_CHARACTERISTICS.HIGH_ENTROPY_VA in binary.optional_header.dll_characteristics_lists
def check_PE_RELOC_SECTION(executable) -> bool:
'''Check for a reloc section. This is required for functional ASLR.'''
binary = lief.parse(executable)
return binary.has_relocations
def check_MACHO_NOUNDEFS(executable) -> bool:
'''
Check for no undefined references.
'''
binary = lief.parse(executable)
return binary.header.has(lief.MachO.HEADER_FLAGS.NOUNDEFS)
def check_MACHO_LAZY_BINDINGS(executable) -> bool:
'''
Check for no lazy bindings.
We don't use or check for MH_BINDATLOAD. See #18295.
'''
binary = lief.parse(executable)
return binary.dyld_info.lazy_bind == (0,0)
def check_MACHO_Canary(executable) -> bool:
'''
Check for use of stack canary
'''
binary = lief.parse(executable)
return binary.has_symbol('___stack_chk_fail')
def check_PIE(executable) -> bool:
'''
Check for position independent executable (PIE),
allowing for address space randomization.
'''
binary = lief.parse(executable)
return binary.is_pie
def check_NX(executable) -> bool:
'''
Check for no stack execution
'''
binary = lief.parse(executable)
return binary.has_nx
def check_control_flow(executable) -> bool:
'''
Check for control flow instrumentation
'''
binary = lief.parse(executable)
content = binary.get_content_from_virtual_address(binary.entrypoint, 4, lief.Binary.VA_TYPES.AUTO)
if content == [243, 15, 30, 250]: # endbr64
return True
return False
CHECKS = {
'ELF': [
('PIE', check_ELF_PIE),
('NX', check_ELF_NX),
('RELRO', check_ELF_RELRO),
('Canary', check_ELF_Canary),
('separate_code', check_ELF_separate_code),
],
'PE': [
('PIE', check_PIE),
('DYNAMIC_BASE', check_PE_DYNAMIC_BASE),
('HIGH_ENTROPY_VA', check_PE_HIGH_ENTROPY_VA),
('NX', check_NX),
('RELOC_SECTION', check_PE_RELOC_SECTION)
],
'MACHO': [
('PIE', check_PIE),
('NOUNDEFS', check_MACHO_NOUNDEFS),
('NX', check_NX),
('LAZY_BINDINGS', check_MACHO_LAZY_BINDINGS),
('Canary', check_MACHO_Canary),
('CONTROL_FLOW', check_control_flow),
]
}
def identify_executable(executable) -> Optional[str]:
with open(filename, 'rb') as f:
magic = f.read(4)
if magic.startswith(b'MZ'):
return 'PE'
elif magic.startswith(b'\x7fELF'):
return 'ELF'
elif magic.startswith(b'\xcf\xfa'):
return 'MACHO'
return None
if __name__ == '__main__':
retval: int = 0
for filename in sys.argv[1:]:
try:
etype = identify_executable(filename)
if etype is None:
print(f'{filename}: unknown format')
retval = 1
continue
failed: List[str] = []
for (name, func) in CHECKS[etype]:
if not func(filename):
failed.append(name)
if failed:
print(f'{filename}: failed {" ".join(failed)}')
retval = 1
except IOError:
print(f'{filename}: cannot open')
retval = 1
sys.exit(retval)

10
contrib/devtools/split-debug.sh.in Normal file
View file

@ -0,0 +1,10 @@
#!/bin/sh
set -e
if [ $# -ne 3 ];
then echo "usage: $0 <input> <stripped-binary> <debug-binary>"
fi
@OBJCOPY@ --enable-deterministic-archives -p --only-keep-debug $1 $3
@OBJCOPY@ --enable-deterministic-archives -p --strip-debug $1 $2
@STRIP@ --enable-deterministic-archives -p -s $2
@OBJCOPY@ --enable-deterministic-archives -p --add-gnu-debuglink=$3 $2

308
contrib/devtools/symbol-check.py Executable file
View file

@ -0,0 +1,308 @@
#!/usr/bin/env python3
# Copyright (c) 2014 Wladimir J. van der Laan
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
A script to check that release executables only contain certain symbols
and are only linked against allowed libraries.
Example usage:
find ../path/to/binaries -type f -executable | xargs python3 contrib/devtools/symbol-check.py
'''
import subprocess
import sys
from typing import List, Optional
import lief
import pixie
from utils import determine_wellknown_cmd
# Debian 8 (Jessie) EOL: 2020. https://wiki.debian.org/DebianReleases#Production_Releases
#
# - g++ version 4.9.2 (https://packages.debian.org/search?suite=jessie&arch=any&searchon=names&keywords=g%2B%2B)
# - libc version 2.19 (https://packages.debian.org/search?suite=jessie&arch=any&searchon=names&keywords=libc6)
#
# Ubuntu 16.04 (Xenial) EOL: 2024. https://wiki.ubuntu.com/Releases
#
# - g++ version 5.3.1 (https://packages.ubuntu.com/search?keywords=g%2B%2B&searchon=names&suite=xenial&section=all)
# - libc version 2.23.0 (https://packages.ubuntu.com/search?keywords=libc6&searchon=names&suite=xenial&section=all)
#
# CentOS 7 EOL: 2024. https://wiki.centos.org/FAQ/General
#
# - g++ version 4.8.5 (http://mirror.centos.org/centos/7/os/x86_64/Packages/)
# - libc version 2.17 (http://mirror.centos.org/centos/7/os/x86_64/Packages/)
#
# Taking the minimum of these as our target.
#
# According to GNU ABI document (https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html) this corresponds to:
# GCC 4.8.5: GCC_4.8.0
# (glibc) GLIBC_2_17
#
MAX_VERSIONS = {
'GCC': (4,8,0),
'GLIBC': {
pixie.EM_386: (2,18),
pixie.EM_X86_64: (2,18),
pixie.EM_ARM: (2,18),
pixie.EM_AARCH64:(2,18),
pixie.EM_PPC64: (2,18),
pixie.EM_RISCV: (2,27),
},
'LIBATOMIC': (1,0),
'V': (0,5,0), # xkb (digibyte-qt only)
}
# See here for a description of _IO_stdin_used:
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261#109
# Ignore symbols that are exported as part of every executable
IGNORE_EXPORTS = {
'_edata', '_end', '__end__', '_init', '__bss_start', '__bss_start__', '_bss_end__', '__bss_end__', '_fini', '_IO_stdin_used', 'stdin', 'stdout', 'stderr',
'environ', '_environ', '__environ',
}
# Allowed NEEDED libraries
ELF_ALLOWED_LIBRARIES = {
# digibyted and digibyte-qt
'libgcc_s.so.1', # GCC base support
'libc.so.6', # C library
'libpthread.so.0', # threading
'libm.so.6', # math library
'librt.so.1', # real-time (clock)
'libatomic.so.1',
'ld-linux-x86-64.so.2', # 64-bit dynamic linker
'ld-linux.so.2', # 32-bit dynamic linker
'ld-linux-aarch64.so.1', # 64-bit ARM dynamic linker
'ld-linux-armhf.so.3', # 32-bit ARM dynamic linker
'ld64.so.1', # POWER64 ABIv1 dynamic linker
'ld64.so.2', # POWER64 ABIv2 dynamic linker
'ld-linux-riscv64-lp64d.so.1', # 64-bit RISC-V dynamic linker
# digibyte-qt only
'libxcb.so.1', # part of X11
'libxkbcommon.so.0', # keyboard keymapping
'libxkbcommon-x11.so.0', # keyboard keymapping
'libfontconfig.so.1', # font support
'libfreetype.so.6', # font parsing
'libdl.so.2', # programming interface to dynamic linker
'libxcb-icccm.so.4',
'libxcb-image.so.0',
'libxcb-shm.so.0',
'libxcb-keysyms.so.1',
'libxcb-randr.so.0',
'libxcb-render-util.so.0',
'libxcb-render.so.0',
'libxcb-shape.so.0',
'libxcb-sync.so.1',
'libxcb-xfixes.so.0',
'libxcb-xinerama.so.0',
'libxcb-xkb.so.1',
}
MACHO_ALLOWED_LIBRARIES = {
# digibyted and digibyte-qt
'libc++.1.dylib', # C++ Standard Library
'libSystem.B.dylib', # libc, libm, libpthread, libinfo
# digibyte-qt only
'AppKit', # user interface
'ApplicationServices', # common application tasks.
'Carbon', # deprecated c back-compat API
'ColorSync',
'CoreFoundation', # low level func, data types
'CoreGraphics', # 2D rendering
'CoreServices', # operating system services
'CoreText', # interface for laying out text and handling fonts.
'CoreVideo', # video processing
'Foundation', # base layer functionality for apps/frameworks
'ImageIO', # read and write image file formats.
'IOKit', # user-space access to hardware devices and drivers.
'IOSurface', # cross process image/drawing buffers
'libobjc.A.dylib', # Objective-C runtime library
'Metal', # 3D graphics
'Security', # access control and authentication
'QuartzCore', # animation
}
PE_ALLOWED_LIBRARIES = {
'ADVAPI32.dll', # security & registry
'IPHLPAPI.DLL', # IP helper API
'KERNEL32.dll', # win32 base APIs
'msvcrt.dll', # C standard library for MSVC
'SHELL32.dll', # shell API
'USER32.dll', # user interface
'WS2_32.dll', # sockets
# digibyte-qt only
'dwmapi.dll', # desktop window manager
'GDI32.dll', # graphics device interface
'IMM32.dll', # input method editor
'NETAPI32.dll',
'ole32.dll', # component object model
'OLEAUT32.dll', # OLE Automation API
'SHLWAPI.dll', # light weight shell API
'USERENV.dll',
'UxTheme.dll',
'VERSION.dll', # version checking
'WINMM.dll', # WinMM audio API
'WTSAPI32.dll',
}
class CPPFilt(object):
'''
Demangle C++ symbol names.
Use a pipe to the 'c++filt' command.
'''
def __init__(self):
self.proc = subprocess.Popen(determine_wellknown_cmd('CPPFILT', 'c++filt'), stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True)
def __call__(self, mangled):
self.proc.stdin.write(mangled + '\n')
self.proc.stdin.flush()
return self.proc.stdout.readline().rstrip()
def close(self):
self.proc.stdin.close()
self.proc.stdout.close()
self.proc.wait()
def check_version(max_versions, version, arch) -> bool:
if '_' in version:
(lib, _, ver) = version.rpartition('_')
else:
lib = version
ver = '0'
ver = tuple([int(x) for x in ver.split('.')])
if not lib in max_versions:
return False
if isinstance(max_versions[lib], tuple):
return ver <= max_versions[lib]
else:
return ver <= max_versions[lib][arch]
def check_imported_symbols(filename) -> bool:
elf = pixie.load(filename)
cppfilt = CPPFilt()
ok: bool = True
for symbol in elf.dyn_symbols:
if not symbol.is_import:
continue
sym = symbol.name.decode()
version = symbol.version.decode() if symbol.version is not None else None
if version and not check_version(MAX_VERSIONS, version, elf.hdr.e_machine):
print('{}: symbol {} from unsupported version {}'.format(filename, cppfilt(sym), version))
ok = False
return ok
def check_exported_symbols(filename) -> bool:
elf = pixie.load(filename)
cppfilt = CPPFilt()
ok: bool = True
for symbol in elf.dyn_symbols:
if not symbol.is_export:
continue
sym = symbol.name.decode()
if elf.hdr.e_machine == pixie.EM_RISCV or sym in IGNORE_EXPORTS:
continue
print('{}: export of symbol {} not allowed'.format(filename, cppfilt(sym)))
ok = False
return ok
def check_ELF_libraries(filename) -> bool:
ok: bool = True
elf = pixie.load(filename)
for library_name in elf.query_dyn_tags(pixie.DT_NEEDED):
assert(isinstance(library_name, bytes))
if library_name.decode() not in ELF_ALLOWED_LIBRARIES:
print('{}: NEEDED library {} is not allowed'.format(filename, library_name.decode()))
ok = False
return ok
def check_MACHO_libraries(filename) -> bool:
ok: bool = True
binary = lief.parse(filename)
for dylib in binary.libraries:
split = dylib.name.split('/')
if split[-1] not in MACHO_ALLOWED_LIBRARIES:
print(f'{split[-1]} is not in ALLOWED_LIBRARIES!')
ok = False
return ok
def check_MACHO_min_os(filename) -> bool:
binary = lief.parse(filename)
if binary.build_version.minos == [10,15,0]:
return True
return False
def check_MACHO_sdk(filename) -> bool:
binary = lief.parse(filename)
if binary.build_version.sdk == [11, 0, 0]:
return True
return False
def check_PE_libraries(filename) -> bool:
ok: bool = True
binary = lief.parse(filename)
for dylib in binary.libraries:
if dylib not in PE_ALLOWED_LIBRARIES:
print(f'{dylib} is not in ALLOWED_LIBRARIES!')
ok = False
return ok
def check_PE_subsystem_version(filename) -> bool:
binary = lief.parse(filename)
major: int = binary.optional_header.major_subsystem_version
minor: int = binary.optional_header.minor_subsystem_version
if major == 6 and minor == 1:
return True
return False
CHECKS = {
'ELF': [
('IMPORTED_SYMBOLS', check_imported_symbols),
('EXPORTED_SYMBOLS', check_exported_symbols),
('LIBRARY_DEPENDENCIES', check_ELF_libraries)
],
'MACHO': [
('DYNAMIC_LIBRARIES', check_MACHO_libraries),
('MIN_OS', check_MACHO_min_os),
('SDK', check_MACHO_sdk),
],
'PE' : [
('DYNAMIC_LIBRARIES', check_PE_libraries),
('SUBSYSTEM_VERSION', check_PE_subsystem_version),
]
}
def identify_executable(executable) -> Optional[str]:
with open(filename, 'rb') as f:
magic = f.read(4)
if magic.startswith(b'MZ'):
return 'PE'
elif magic.startswith(b'\x7fELF'):
return 'ELF'
elif magic.startswith(b'\xcf\xfa'):
return 'MACHO'
return None
if __name__ == '__main__':
retval: int = 0
for filename in sys.argv[1:]:
try:
etype = identify_executable(filename)
if etype is None:
print(f'{filename}: unknown format')
retval = 1
continue
failed: List[str] = []
for (name, func) in CHECKS[etype]:
if not func(filename):
failed.append(name)
if failed:
print(f'{filename}: failed {" ".join(failed)}')
retval = 1
except IOError:
print(f'{filename}: cannot open')
retval = 1
sys.exit(retval)

112
contrib/devtools/test-security-check.py Executable file
View file

@ -0,0 +1,112 @@
#!/usr/bin/env python3
# Copyright (c) 2015-2021 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Test script for security-check.py
'''
import lief #type:ignore
import os
import subprocess
from typing import List
import unittest
from utils import determine_wellknown_cmd
def write_testcode(filename):
with open(filename, 'w', encoding="utf8") as f:
f.write('''
#include <stdio.h>
int main()
{
printf("the quick brown fox jumps over the lazy god\\n");
return 0;
}
''')
def clean_files(source, executable):
os.remove(source)
os.remove(executable)
def call_security_check(cc, source, executable, options):
subprocess.run([*cc,source,'-o',executable] + options, check=True)
p = subprocess.run(['./contrib/devtools/security-check.py',executable], stdout=subprocess.PIPE, universal_newlines=True)
return (p.returncode, p.stdout.rstrip())
def get_arch(cc, source, executable):
subprocess.run([*cc, source, '-o', executable], check=True)
binary = lief.parse(executable)
arch = binary.abstract.header.architecture
os.remove(executable)
return arch
class TestSecurityChecks(unittest.TestCase):
def test_ELF(self):
source = 'test1.c'
executable = 'test1'
cc = determine_wellknown_cmd('CC', 'gcc')
write_testcode(source)
arch = get_arch(cc, source, executable)
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-zexecstack','-fno-stack-protector','-Wl,-znorelro','-no-pie','-fno-PIE', '-Wl,-z,separate-code']),
(1, executable+': failed PIE NX RELRO Canary'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fno-stack-protector','-Wl,-znorelro','-no-pie','-fno-PIE', '-Wl,-z,separate-code']),
(1, executable+': failed PIE RELRO Canary'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-znorelro','-no-pie','-fno-PIE', '-Wl,-z,separate-code']),
(1, executable+': failed PIE RELRO'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-znorelro','-pie','-fPIE', '-Wl,-z,separate-code']),
(1, executable+': failed RELRO'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-zrelro','-Wl,-z,now','-pie','-fPIE', '-Wl,-z,noseparate-code']),
(1, executable+': failed separate_code'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-zrelro','-Wl,-z,now','-pie','-fPIE', '-Wl,-z,separate-code']),
(0, ''))
clean_files(source, executable)
def test_PE(self):
source = 'test1.c'
executable = 'test1.exe'
cc = determine_wellknown_cmd('CC', 'x86_64-w64-mingw32-gcc')
write_testcode(source)
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--no-nxcompat','-Wl,--disable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-no-pie','-fno-PIE']),
(1, executable+': failed PIE DYNAMIC_BASE HIGH_ENTROPY_VA NX RELOC_SECTION'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--disable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-no-pie','-fno-PIE']),
(1, executable+': failed PIE DYNAMIC_BASE HIGH_ENTROPY_VA RELOC_SECTION'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--enable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-no-pie','-fno-PIE']),
(1, executable+': failed PIE DYNAMIC_BASE HIGH_ENTROPY_VA'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--enable-reloc-section','-Wl,--no-dynamicbase','-Wl,--no-high-entropy-va','-pie','-fPIE']),
(1, executable+': failed PIE DYNAMIC_BASE HIGH_ENTROPY_VA')) # -pie -fPIE does nothing unless --dynamicbase is also supplied
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--enable-reloc-section','-Wl,--dynamicbase','-Wl,--no-high-entropy-va','-pie','-fPIE']),
(1, executable+': failed HIGH_ENTROPY_VA'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--enable-reloc-section','-Wl,--dynamicbase','-Wl,--high-entropy-va','-pie','-fPIE']),
(0, ''))
clean_files(source, executable)
def test_MACHO(self):
source = 'test1.c'
executable = 'test1'
cc = determine_wellknown_cmd('CC', 'clang')
write_testcode(source)
arch = get_arch(cc, source, executable)
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector']),
(1, executable+': failed PIE NOUNDEFS NX LAZY_BINDINGS Canary CONTROL_FLOW'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fstack-protector-all']),
(1, executable+': failed PIE NOUNDEFS NX LAZY_BINDINGS CONTROL_FLOW'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-fstack-protector-all']),
(1, executable+': failed PIE NOUNDEFS LAZY_BINDINGS CONTROL_FLOW'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all']),
(1, executable+': failed PIE LAZY_BINDINGS CONTROL_FLOW'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all']),
(1, executable+': failed PIE CONTROL_FLOW'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
(1, executable+': failed PIE'))
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
(0, ''))
clean_files(source, executable)
if __name__ == '__main__':
unittest.main()

204
contrib/devtools/test-symbol-check.py Executable file
View file

@ -0,0 +1,204 @@
#!/usr/bin/env python3
# Copyright (c) 2020-2021 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Test script for symbol-check.py
'''
import os
import subprocess
from typing import List
import unittest
from utils import determine_wellknown_cmd
def call_symbol_check(cc: List[str], source, executable, options):
# This should behave the same as AC_TRY_LINK, so arrange well-known flags
# in the same order as autoconf would.
#
# See the definitions for ac_link in autoconf's lib/autoconf/c.m4 file for
# reference.
env_flags: List[str] = []
for var in ['CFLAGS', 'CPPFLAGS', 'LDFLAGS']:
env_flags += filter(None, os.environ.get(var, '').split(' '))
subprocess.run([*cc,source,'-o',executable] + env_flags + options, check=True)
p = subprocess.run(['./contrib/devtools/symbol-check.py',executable], stdout=subprocess.PIPE, universal_newlines=True)
os.remove(source)
os.remove(executable)
return (p.returncode, p.stdout.rstrip())
def get_machine(cc: List[str]):
p = subprocess.run([*cc,'-dumpmachine'], stdout=subprocess.PIPE, universal_newlines=True)
return p.stdout.rstrip()
class TestSymbolChecks(unittest.TestCase):
def test_ELF(self):
source = 'test1.c'
executable = 'test1'
cc = determine_wellknown_cmd('CC', 'gcc')
# there's no way to do this test for RISC-V at the moment; we build for
# RISC-V in a glibc 2.27 environment and we allow all symbols from 2.27.
if 'riscv' in get_machine(cc):
self.skipTest("test not available for RISC-V")
# nextup was introduced in GLIBC 2.24, so is newer than our supported
# glibc (2.18), and available in our release build environment (2.24).
with open(source, 'w', encoding="utf8") as f:
f.write('''
#define _GNU_SOURCE
#include <math.h>
double nextup(double x);
int main()
{
nextup(3.14);
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, ['-lm']),
(1, executable + ': symbol nextup from unsupported version GLIBC_2.24\n' +
executable + ': failed IMPORTED_SYMBOLS'))
# -lutil is part of the libc6 package so a safe bet that it's installed
# it's also out of context enough that it's unlikely to ever become a real dependency
source = 'test2.c'
executable = 'test2'
with open(source, 'w', encoding="utf8") as f:
f.write('''
#include <utmp.h>
int main()
{
login(0);
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, ['-lutil']),
(1, executable + ': NEEDED library libutil.so.1 is not allowed\n' +
executable + ': failed LIBRARY_DEPENDENCIES'))
# finally, check a simple conforming binary
source = 'test3.c'
executable = 'test3'
with open(source, 'w', encoding="utf8") as f:
f.write('''
#include <stdio.h>
int main()
{
printf("42");
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, []),
(0, ''))
def test_MACHO(self):
source = 'test1.c'
executable = 'test1'
cc = determine_wellknown_cmd('CC', 'clang')
with open(source, 'w', encoding="utf8") as f:
f.write('''
#include <expat.h>
int main()
{
XML_ExpatVersion();
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, ['-lexpat', '-Wl,-platform_version','-Wl,macos', '-Wl,11.4', '-Wl,11.4']),
(1, 'libexpat.1.dylib is not in ALLOWED_LIBRARIES!\n' +
f'{executable}: failed DYNAMIC_LIBRARIES MIN_OS SDK'))
source = 'test2.c'
executable = 'test2'
with open(source, 'w', encoding="utf8") as f:
f.write('''
#include <CoreGraphics/CoreGraphics.h>
int main()
{
CGMainDisplayID();
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, ['-framework', 'CoreGraphics', '-Wl,-platform_version','-Wl,macos', '-Wl,11.4', '-Wl,11.4']),
(1, f'{executable}: failed MIN_OS SDK'))
source = 'test3.c'
executable = 'test3'
with open(source, 'w', encoding="utf8") as f:
f.write('''
int main()
{
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, ['-Wl,-platform_version','-Wl,macos', '-Wl,10.15', '-Wl,11.4']),
(1, f'{executable}: failed SDK'))
def test_PE(self):
source = 'test1.c'
executable = 'test1.exe'
cc = determine_wellknown_cmd('CC', 'x86_64-w64-mingw32-gcc')
with open(source, 'w', encoding="utf8") as f:
f.write('''
#include <pdh.h>
int main()
{
PdhConnectMachineA(NULL);
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, ['-lpdh', '-Wl,--major-subsystem-version', '-Wl,6', '-Wl,--minor-subsystem-version', '-Wl,1']),
(1, 'pdh.dll is not in ALLOWED_LIBRARIES!\n' +
executable + ': failed DYNAMIC_LIBRARIES'))
source = 'test2.c'
executable = 'test2.exe'
with open(source, 'w', encoding="utf8") as f:
f.write('''
int main()
{
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, ['-Wl,--major-subsystem-version', '-Wl,9', '-Wl,--minor-subsystem-version', '-Wl,9']),
(1, executable + ': failed SUBSYSTEM_VERSION'))
source = 'test3.c'
executable = 'test3.exe'
with open(source, 'w', encoding="utf8") as f:
f.write('''
#include <combaseapi.h>
int main()
{
CoFreeUnusedLibrariesEx(0,0);
return 0;
}
''')
self.assertEqual(call_symbol_check(cc, source, executable, ['-lole32', '-Wl,--major-subsystem-version', '-Wl,6', '-Wl,--minor-subsystem-version', '-Wl,1']),
(0, ''))
if __name__ == '__main__':
unittest.main()

151
contrib/devtools/test_deterministic_coverage.sh Executable file
View file

@ -0,0 +1,151 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019-2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Test for deterministic coverage across unit test runs.
export LC_ALL=C
# Use GCOV_EXECUTABLE="gcov" if compiling with gcc.
# Use GCOV_EXECUTABLE="llvm-cov gcov" if compiling with clang.
GCOV_EXECUTABLE="gcov"
# Disable tests known to cause non-deterministic behaviour and document the source or point of non-determinism.
NON_DETERMINISTIC_TESTS=(
"blockfilter_index_tests/blockfilter_index_initial_sync" # src/checkqueue.h: In CCheckQueue::Loop(): while (queue.empty()) { ... }
"coinselector_tests/knapsack_solver_test" # coinselector_tests.cpp: if (equal_sets(setCoinsRet, setCoinsRet2))
"fs_tests/fsbridge_fstream" # deterministic test failure?
"miner_tests/CreateNewBlock_validity" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"scheduler_tests/manythreads" # scheduler.cpp: CScheduler::serviceQueue()
"scheduler_tests/singlethreadedscheduler_ordered" # scheduler.cpp: CScheduler::serviceQueue()
"txvalidationcache_tests/checkinputs_test" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"txvalidationcache_tests/tx_mempool_block_doublespend" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"txindex_tests/txindex_initial_sync" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"txvalidation_tests/tx_mempool_reject_coinbase" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"validation_block_tests/processnewblock_signals_ordering" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"wallet_tests/coin_mark_dirty_immature_credit" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"wallet_tests/dummy_input_size_test" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"wallet_tests/importmulti_rescan" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"wallet_tests/importwallet_rescan" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"wallet_tests/ListCoins" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"wallet_tests/scan_for_wallet_transactions" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
"wallet_tests/wallet_disableprivkeys" # validation.cpp: if (GetMainSignals().CallbacksPending() > 10)
)
TEST_DIGIBYTE_BINARY="src/test/test_digibyte"
print_usage() {
echo "Usage: $0 [custom test filter (default: all but known non-deterministic tests)] [number of test runs (default: 2)]"
}
N_TEST_RUNS=2
BOOST_TEST_RUN_FILTERS=""
if [[ $# != 0 ]]; then
if [[ $1 == "--help" ]]; then
print_usage
exit
fi
PARSED_ARGUMENTS=0
if [[ $1 =~ [a-z] ]]; then
BOOST_TEST_RUN_FILTERS=$1
PARSED_ARGUMENTS=$((PARSED_ARGUMENTS + 1))
shift
fi
if [[ $1 =~ ^[0-9]+$ ]]; then
N_TEST_RUNS=$1
PARSED_ARGUMENTS=$((PARSED_ARGUMENTS + 1))
shift
fi
if [[ ${PARSED_ARGUMENTS} == 0 || $# -gt 2 || ${N_TEST_RUNS} -lt 2 ]]; then
print_usage
exit
fi
fi
if [[ ${BOOST_TEST_RUN_FILTERS} == "" ]]; then
BOOST_TEST_RUN_FILTERS="$(IFS=":"; echo "!${NON_DETERMINISTIC_TESTS[*]}" | sed 's/:/:!/g')"
else
echo "Using Boost test filter: ${BOOST_TEST_RUN_FILTERS}"
echo
fi
if ! command -v gcov > /dev/null; then
echo "Error: gcov not installed. Exiting."
exit 1
fi
if ! command -v gcovr > /dev/null; then
echo "Error: gcovr not installed. Exiting."
exit 1
fi
if [[ ! -e ${TEST_DIGIBYTE_BINARY} ]]; then
echo "Error: Executable ${TEST_DIGIBYTE_BINARY} not found. Run \"./configure --enable-lcov\" and compile."
exit 1
fi
get_file_suffix_count() {
find src/ -type f -name "*.$1" | wc -l
}
if [[ $(get_file_suffix_count gcno) == 0 ]]; then
echo "Error: Could not find any *.gcno files. The *.gcno files are generated by the compiler. Run \"./configure --enable-lcov\" and re-compile."
exit 1
fi
get_covr_filename() {
echo "gcovr.run-$1.txt"
}
TEST_RUN_ID=0
while [[ ${TEST_RUN_ID} -lt ${N_TEST_RUNS} ]]; do
TEST_RUN_ID=$((TEST_RUN_ID + 1))
echo "[$(date +"%Y-%m-%d %H:%M:%S")] Measuring coverage, run #${TEST_RUN_ID} of ${N_TEST_RUNS}"
find src/ -type f -name "*.gcda" -exec rm {} \;
if [[ $(get_file_suffix_count gcda) != 0 ]]; then
echo "Error: Stale *.gcda files found. Exiting."
exit 1
fi
TEST_OUTPUT_TEMPFILE=$(mktemp)
if ! BOOST_TEST_RUN_FILTERS="${BOOST_TEST_RUN_FILTERS}" ${TEST_DIGIBYTE_BINARY} > "${TEST_OUTPUT_TEMPFILE}" 2>&1; then
cat "${TEST_OUTPUT_TEMPFILE}"
rm "${TEST_OUTPUT_TEMPFILE}"
exit 1
fi
rm "${TEST_OUTPUT_TEMPFILE}"
if [[ $(get_file_suffix_count gcda) == 0 ]]; then
echo "Error: Running the test suite did not create any *.gcda files. The gcda files are generated when the instrumented test programs are executed. Run \"./configure --enable-lcov\" and re-compile."
exit 1
fi
GCOVR_TEMPFILE=$(mktemp)
if ! gcovr --gcov-executable "${GCOV_EXECUTABLE}" -r src/ > "${GCOVR_TEMPFILE}"; then
echo "Error: gcovr failed. Output written to ${GCOVR_TEMPFILE}. Exiting."
exit 1
fi
GCOVR_FILENAME=$(get_covr_filename ${TEST_RUN_ID})
mv "${GCOVR_TEMPFILE}" "${GCOVR_FILENAME}"
if grep -E "^TOTAL *0 *0 " "${GCOVR_FILENAME}"; then
echo "Error: Spurious gcovr output. Make sure the correct GCOV_EXECUTABLE variable is set in $0 (\"gcov\" for gcc, \"llvm-cov gcov\" for clang)."
exit 1
fi
if [[ ${TEST_RUN_ID} != 1 ]]; then
COVERAGE_DIFF=$(diff -u "$(get_covr_filename 1)" "${GCOVR_FILENAME}")
if [[ ${COVERAGE_DIFF} != "" ]]; then
echo
echo "The line coverage is non-deterministic between runs. Exiting."
echo
echo "The test suite must be deterministic in the sense that the set of lines executed at least"
echo "once must be identical between runs. This is a necessary condition for meaningful"
echo "coverage measuring."
echo
echo "${COVERAGE_DIFF}"
exit 1
fi
rm "${GCOVR_FILENAME}"
fi
done
echo
echo "Coverage test passed: Deterministic coverage across ${N_TEST_RUNS} runs."
exit

215
contrib/devtools/update-translations.py Executable file
View file

@ -0,0 +1,215 @@
#!/usr/bin/env python3
# Copyright (c) 2014 Wladimir J. van der Laan
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Run this script from the root of the repository to update all translations from
transifex.
It will do the following automatically:
- fetch all translations using the tx tool
- post-process them into valid and committable format
- remove invalid control characters
- remove location tags (makes diffs less noisy)
TODO:
- auto-add new translations to the build system according to the translation process
'''
import subprocess
import re
import sys
import os
import io
import xml.etree.ElementTree as ET
# Name of transifex tool
TX = 'tx'
# Name of source language file
SOURCE_LANG = 'digibyte_en.ts'
# Directory with locale files
LOCALE_DIR = 'src/qt/locale'
# Minimum number of messages for translation to be considered at all
MIN_NUM_MESSAGES = 10
# Regexp to check for DigiByte addresses
ADDRESS_REGEXP = re.compile('([13]|bc1)[a-zA-Z0-9]{30,}')
def check_at_repository_root():
if not os.path.exists('.git'):
print('No .git directory found')
print('Execute this script at the root of the repository', file=sys.stderr)
sys.exit(1)
def fetch_all_translations():
if subprocess.call([TX, 'pull', '-f', '-a']):
print('Error while fetching translations', file=sys.stderr)
sys.exit(1)
def find_format_specifiers(s):
'''Find all format specifiers in a string.'''
pos = 0
specifiers = []
while True:
percent = s.find('%', pos)
if percent < 0:
break
specifiers.append(s[percent+1])
pos = percent+2
return specifiers
def split_format_specifiers(specifiers):
'''Split format specifiers between numeric (Qt) and others (strprintf)'''
numeric = []
other = []
for s in specifiers:
if s in {'1','2','3','4','5','6','7','8','9'}:
numeric.append(s)
else:
other.append(s)
# If both numeric format specifiers and "others" are used, assume we're dealing
# with a Qt-formatted message. In the case of Qt formatting (see https://doc.qt.io/qt-5/qstring.html#arg)
# only numeric formats are replaced at all. This means "(percentage: %1%)" is valid, without needing
# any kind of escaping that would be necessary for strprintf. Without this, this function
# would wrongly detect '%)' as a printf format specifier.
if numeric:
other = []
# numeric (Qt) can be present in any order, others (strprintf) must be in specified order
return set(numeric),other
def sanitize_string(s):
'''Sanitize string for printing'''
return s.replace('\n',' ')
def check_format_specifiers(source, translation, errors, numerus):
source_f = split_format_specifiers(find_format_specifiers(source))
# assert that no source messages contain both Qt and strprintf format specifiers
# if this fails, go change the source as this is hacky and confusing!
assert(not(source_f[0] and source_f[1]))
try:
translation_f = split_format_specifiers(find_format_specifiers(translation))
except IndexError:
errors.append("Parse error in translation for '%s': '%s'" % (sanitize_string(source), sanitize_string(translation)))
return False
else:
if source_f != translation_f:
if numerus and source_f == (set(), ['n']) and translation_f == (set(), []) and translation.find('%') == -1:
# Allow numerus translations to omit %n specifier (usually when it only has one possible value)
return True
errors.append("Mismatch between '%s' and '%s'" % (sanitize_string(source), sanitize_string(translation)))
return False
return True
def all_ts_files(suffix=''):
for filename in os.listdir(LOCALE_DIR):
# process only language files, and do not process source language
if not filename.endswith('.ts'+suffix) or filename == SOURCE_LANG+suffix:
continue
if suffix: # remove provided suffix
filename = filename[0:-len(suffix)]
filepath = os.path.join(LOCALE_DIR, filename)
yield(filename, filepath)
FIX_RE = re.compile(b'[\x00-\x09\x0b\x0c\x0e-\x1f]')
def remove_invalid_characters(s):
'''Remove invalid characters from translation string'''
return FIX_RE.sub(b'', s)
# Override cdata escape function to make our output match Qt's (optional, just for cleaner diffs for
# comparison, disable by default)
_orig_escape_cdata = None
def escape_cdata(text):
text = _orig_escape_cdata(text)
text = text.replace("'", '&apos;')
text = text.replace('"', '&quot;')
return text
def contains_digibyte_addr(text, errors):
if text != None and ADDRESS_REGEXP.search(text) != None:
errors.append('Translation "%s" contains a digibyte address. This will be removed.' % (text))
return True
return False
def postprocess_translations(reduce_diff_hacks=False):
print('Checking and postprocessing...')
if reduce_diff_hacks:
global _orig_escape_cdata
_orig_escape_cdata = ET._escape_cdata
ET._escape_cdata = escape_cdata
for (filename,filepath) in all_ts_files():
os.rename(filepath, filepath+'.orig')
have_errors = False
for (filename,filepath) in all_ts_files('.orig'):
# pre-fixups to cope with transifex output
parser = ET.XMLParser(encoding='utf-8') # need to override encoding because 'utf8' is not understood only 'utf-8'
with open(filepath + '.orig', 'rb') as f:
data = f.read()
# remove control characters; this must be done over the entire file otherwise the XML parser will fail
data = remove_invalid_characters(data)
tree = ET.parse(io.BytesIO(data), parser=parser)
# iterate over all messages in file
root = tree.getroot()
for context in root.findall('context'):
for message in context.findall('message'):
numerus = message.get('numerus') == 'yes'
source = message.find('source').text
translation_node = message.find('translation')
# pick all numerusforms
if numerus:
translations = [i.text for i in translation_node.findall('numerusform')]
else:
translations = [translation_node.text]
for translation in translations:
if translation is None:
continue
errors = []
valid = check_format_specifiers(source, translation, errors, numerus) and not contains_digibyte_addr(translation, errors)
for error in errors:
print('%s: %s' % (filename, error))
if not valid: # set type to unfinished and clear string if invalid
translation_node.clear()
translation_node.set('type', 'unfinished')
have_errors = True
# Remove location tags
for location in message.findall('location'):
message.remove(location)
# Remove entire message if it is an unfinished translation
if translation_node.get('type') == 'unfinished':
context.remove(message)
# check if document is (virtually) empty, and remove it if so
num_messages = 0
for context in root.findall('context'):
for message in context.findall('message'):
num_messages += 1
if num_messages < MIN_NUM_MESSAGES:
print('Removing %s, as it contains only %i messages' % (filepath, num_messages))
continue
# write fixed-up tree
# if diff reduction requested, replace some XML to 'sanitize' to qt formatting
if reduce_diff_hacks:
out = io.BytesIO()
tree.write(out, encoding='utf-8')
out = out.getvalue()
out = out.replace(b' />', b'/>')
with open(filepath, 'wb') as f:
f.write(out)
else:
tree.write(filepath, encoding='utf-8')
return have_errors
if __name__ == '__main__':
check_at_repository_root()
fetch_all_translations()
postprocess_translations()

22
contrib/devtools/utils.py Executable file
View file

@ -0,0 +1,22 @@
#!/usr/bin/env python3
# Copyright (c) 2021 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
'''
Common utility functions
'''
import shutil
import sys
import os
from typing import List
def determine_wellknown_cmd(envvar, progname) -> List[str]:
maybe_env = os.getenv(envvar)
maybe_which = shutil.which(progname)
if maybe_env:
return maybe_env.split(' ') # Well-known vars are often meant to be word-split
elif maybe_which:
return [ maybe_which ]
else:
sys.exit(f"{progname} not found")

44
contrib/devtools/utxo_snapshot.sh Executable file
View file

@ -0,0 +1,44 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
export LC_ALL=C
set -ueo pipefail
if (( $# < 3 )); then
echo 'Usage: utxo_snapshot.sh <generate-at-height> <snapshot-out-path> <digibyte-cli-call ...>'
echo
echo " if <snapshot-out-path> is '-', don't produce a snapshot file but instead print the "
echo " expected assumeutxo hash"
echo
echo 'Examples:'
echo
echo " ./contrib/devtools/utxo_snapshot.sh 570000 utxo.dat ./src/digibyte-cli -datadir=\$(pwd)/testdata"
echo ' ./contrib/devtools/utxo_snapshot.sh 570000 - ./src/digibyte-cli'
exit 1
fi
GENERATE_AT_HEIGHT="${1}"; shift;
OUTPUT_PATH="${1}"; shift;
# Most of the calls we make take a while to run, so pad with a lengthy timeout.
DIGIBYTE_CLI_CALL="${*} -rpcclienttimeout=9999999"
# Block we'll invalidate/reconsider to rewind/fast-forward the chain.
PIVOT_BLOCKHASH=$($DIGIBYTE_CLI_CALL getblockhash $(( GENERATE_AT_HEIGHT + 1 )) )
(>&2 echo "Rewinding chain back to height ${GENERATE_AT_HEIGHT} (by invalidating ${PIVOT_BLOCKHASH}); this may take a while")
${DIGIBYTE_CLI_CALL} invalidateblock "${PIVOT_BLOCKHASH}"
if [[ "${OUTPUT_PATH}" = "-" ]]; then
(>&2 echo "Generating txoutset info...")
${DIGIBYTE_CLI_CALL} gettxoutsetinfo | grep hash_serialized_2 | sed 's/^.*: "\(.\+\)\+",/\1/g'
else
(>&2 echo "Generating UTXO snapshot...")
${DIGIBYTE_CLI_CALL} dumptxoutset "${OUTPUT_PATH}"
fi
(>&2 echo "Restoring chain to original height; this may take a while")
${DIGIBYTE_CLI_CALL} reconsiderblock "${PIVOT_BLOCKHASH}"

142
contrib/digibyte-cli.bash-completion Normal file
View file

@ -0,0 +1,142 @@
# bash programmable completion for digibyte-cli(1)
# Copyright (c) 2012-2016 The Bitcoin Core developers
# Copyright (c) 2012-2019 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
# call $digibyte-cli for RPC
_digibyte_rpc() {
# determine already specified args necessary for RPC
local rpcargs=()
for i in ${COMP_LINE}; do
case "$i" in
-conf=*|-datadir=*|-regtest|-rpc*|-testnet)
rpcargs=( "${rpcargs[@]}" "$i" )
;;
esac
done
$digibyte_cli "${rpcargs[@]}" "$@"
}
_digibyte_cli() {
local cur prev words=() cword
local digibyte_cli
# save and use original argument to invoke digibyte-cli for -help, help and RPC
# as digibyte-cli might not be in $PATH
digibyte_cli="$1"
COMPREPLY=()
_get_comp_words_by_ref -n = cur prev words cword
if ((cword > 5)); then
case ${words[cword-5]} in
sendtoaddress)
COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
return 0
;;
esac
fi
if ((cword > 4)); then
case ${words[cword-4]} in
importaddress|listtransactions|setban)
COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
return 0
;;
signrawtransactionwithkey|signrawtransactionwithwallet)
COMPREPLY=( $( compgen -W "ALL NONE SINGLE ALL|ANYONECANPAY NONE|ANYONECANPAY SINGLE|ANYONECANPAY" -- "$cur" ) )
return 0
;;
esac
fi
if ((cword > 3)); then
case ${words[cword-3]} in
addmultisigaddress)
return 0
;;
getbalance|gettxout|importaddress|importpubkey|importprivkey|listreceivedbyaddress|listsinceblock)
COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
return 0
;;
esac
fi
if ((cword > 2)); then
case ${words[cword-2]} in
addnode)
COMPREPLY=( $( compgen -W "add remove onetry" -- "$cur" ) )
return 0
;;
setban)
COMPREPLY=( $( compgen -W "add remove" -- "$cur" ) )
return 0
;;
fundrawtransaction|getblock|getblockheader|getmempoolancestors|getmempooldescendants|getrawtransaction|gettransaction|listreceivedbyaddress|sendrawtransaction)
COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
return 0
;;
esac
fi
case "$prev" in
backupwallet|dumpwallet|importwallet)
_filedir
return 0
;;
getaddednodeinfo|getrawmempool|lockunspent)
COMPREPLY=( $( compgen -W "true false" -- "$cur" ) )
return 0
;;
getbalance|getnewaddress|listtransactions|sendmany)
return 0
;;
esac
case "$cur" in
-conf=*)
cur="${cur#*=}"
_filedir
return 0
;;
-datadir=*)
cur="${cur#*=}"
_filedir -d
return 0
;;
-*=*) # prevent nonsense completions
return 0
;;
*)
local helpopts commands
# only parse -help if senseful
if [[ -z "$cur" || "$cur" =~ ^- ]]; then
helpopts=$($digibyte_cli -help 2>&1 | awk '$1 ~ /^-/ { sub(/=.*/, "="); print $1 }' )
fi
# only parse help if senseful
if [[ -z "$cur" || "$cur" =~ ^[a-z] ]]; then
commands=$(_digibyte_rpc help 2>/dev/null | awk '$1 ~ /^[a-z]/ { print $1; }')
fi
COMPREPLY=( $( compgen -W "$helpopts $commands" -- "$cur" ) )
# Prevent space if an argument is desired
if [[ $COMPREPLY == *= ]]; then
compopt -o nospace
fi
return 0
;;
esac
} &&
complete -F _digibyte_cli digibyte-cli
# Local variables:
# mode: shell-script
# sh-basic-offset: 4
# sh-indent-comment: t
# indent-tabs-mode: nil
# End:
# ex: ts=4 sw=4 et filetype=sh

21
contrib/digibyte-qt.pro Normal file
View file

@ -0,0 +1,21 @@
FORMS += \
../src/qt/forms/aboutdialog.ui \
../src/qt/forms/addressbookpage.ui \
../src/qt/forms/askpassphrasedialog.ui \
../src/qt/forms/coincontroldialog.ui \
../src/qt/forms/editaddressdialog.ui \
../src/qt/forms/helpmessagedialog.ui \
../src/qt/forms/intro.ui \
../src/qt/forms/openuridialog.ui \
../src/qt/forms/optionsdialog.ui \
../src/qt/forms/overviewpage.ui \
../src/qt/forms/receivecoinsdialog.ui \
../src/qt/forms/receiverequestdialog.ui \
../src/qt/forms/debugwindow.ui \
../src/qt/forms/sendcoinsdialog.ui \
../src/qt/forms/sendcoinsentry.ui \
../src/qt/forms/signverifymessagedialog.ui \
../src/qt/forms/transactiondescdialog.ui \
RESOURCES += \
../src/qt/digibyte.qrc

57
contrib/digibyte-tx.bash-completion Normal file
View file

@ -0,0 +1,57 @@
# bash programmable completion for digibyte-tx(1)
# Copyright (c) 2016 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
_digibyte_tx() {
local cur prev words=() cword
local digibyte_tx
# save and use original argument to invoke digibyte-tx for -help
# it might not be in $PATH
digibyte_tx="$1"
COMPREPLY=()
_get_comp_words_by_ref -n =: cur prev words cword
case "$cur" in
load=*:*)
cur="${cur#load=*:}"
_filedir
return 0
;;
*=*) # prevent attempts to complete other arguments
return 0
;;
esac
if [[ "$cword" == 1 || ( "$prev" != "-create" && "$prev" == -* ) ]]; then
# only options (or an uncompletable hex-string) allowed
# parse digibyte-tx -help for options
local helpopts
helpopts=$($digibyte_tx -help | sed -e '/^ -/ p' -e d )
COMPREPLY=( $( compgen -W "$helpopts" -- "$cur" ) )
else
# only commands are allowed
# parse -help for commands
local helpcmds
helpcmds=$($digibyte_tx -help | sed -e '1,/Commands:/d' -e 's/=.*/=/' -e '/^ [a-z]/ p' -e d )
COMPREPLY=( $( compgen -W "$helpcmds" -- "$cur" ) )
fi
# Prevent space if an argument is desired
if [[ $COMPREPLY == *= ]]; then
compopt -o nospace
fi
return 0
} &&
complete -F _digibyte_tx digibyte-tx
# Local variables:
# mode: shell-script
# sh-basic-offset: 4
# sh-indent-comment: t
# indent-tabs-mode: nil
# End:
# ex: ts=4 sw=4 et filetype=sh

57
contrib/digibyted.bash-completion Normal file
View file

@ -0,0 +1,57 @@
# bash programmable completion for digibyted(1) and digibyte-qt(1)
# Copyright (c) 2012-2016 The Bitcoin Core developers
# Copyright (c) 2012-2019 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
_digibyted() {
local cur prev words=() cword
local digibyted
# save and use original argument to invoke digibyted for -help
# it might not be in $PATH
digibyted="$1"
COMPREPLY=()
_get_comp_words_by_ref -n = cur prev words cword
case "$cur" in
-conf=*|-pid=*|-loadblock=*|-rpccookiefile=*|-wallet=*)
cur="${cur#*=}"
_filedir
return 0
;;
-datadir=*)
cur="${cur#*=}"
_filedir -d
return 0
;;
-*=*) # prevent nonsense completions
return 0
;;
*)
# only parse -help if sensible
if [[ -z "$cur" || "$cur" =~ ^- ]]; then
local helpopts
helpopts=$($digibyted -help 2>&1 | awk '$1 ~ /^-/ { sub(/=.*/, "="); print $1 }' )
COMPREPLY=( $( compgen -W "$helpopts" -- "$cur" ) )
fi
# Prevent space if an argument is desired
if [[ $COMPREPLY == *= ]]; then
compopt -o nospace
fi
return 0
;;
esac
} &&
complete -F _digibyted digibyted digibyte-qt
# Local variables:
# mode: shell-script
# sh-basic-offset: 4
# sh-indent-comment: t
# indent-tabs-mode: nil
# End:
# ex: ts=4 sw=4 et filetype=sh

28
contrib/filter-lcov.py Executable file
View file

@ -0,0 +1,28 @@
#!/usr/bin/env python3
# Copyright (c) 2017-2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
import argparse
parser = argparse.ArgumentParser(description='Remove the coverage data from a tracefile for all files matching the pattern.')
parser.add_argument('--pattern', '-p', action='append', help='the pattern of files to remove', required=True)
parser.add_argument('tracefile', help='the tracefile to remove the coverage data from')
parser.add_argument('outfile', help='filename for the output to be written to')
args = parser.parse_args()
tracefile = args.tracefile
pattern = args.pattern
outfile = args.outfile
in_remove = False
with open(tracefile, 'r', encoding="utf8") as f:
with open(outfile, 'w', encoding="utf8") as wf:
for line in f:
for p in pattern:
if line.startswith("SF:") and p in line:
in_remove = True
if not in_remove:
wf.write(line)
if line == 'end_of_record\n':
in_remove = False

112
contrib/genesis/genesis.py Executable file
View file

@ -0,0 +1,112 @@
#!/usr/bin/env python3
import hashlib, struct, time
COIN = 100000000
pszTimestamp = b"Vexta Signet Genesis Block 09/Jul/2026 - VTX signet network begins"
nTime = 1783555202
nBits = 0x1e0377ae
nVersion = 1
genesisReward = 50 * COIN
def sha256d(b):
return hashlib.sha256(hashlib.sha256(b).digest()).digest()
def ser_uint256_zero():
return b"\x00" * 32
def ser_compact_size(n):
if n < 253:
return struct.pack("<B", n)
if n <= 0xffff:
return b"\xfd" + struct.pack("<H", n)
if n <= 0xffffffff:
return b"\xfe" + struct.pack("<I", n)
return b"\xff" + struct.pack("<Q", n)
def push_data(data):
return bytes([len(data)]) + data
def script_num(n):
if n == 0:
return b""
result = bytearray()
neg = n < 0
absvalue = -n if neg else n
while absvalue:
result.append(absvalue & 0xff)
absvalue >>= 8
if result[-1] & 0x80:
result.append(0x80 if neg else 0)
elif neg:
result[-1] |= 0x80
return bytes(result)
def push_num(n):
b = script_num(n)
return bytes([len(b)]) + b
scriptSig = push_num(486604799) + push_num(4) + push_data(pszTimestamp)
scriptPubKey = b"\x00\xac"
tx = b""
tx += struct.pack("<i", 1)
tx += ser_compact_size(1)
tx += ser_uint256_zero()
tx += struct.pack("<I", 0xffffffff)
tx += ser_compact_size(len(scriptSig))
tx += scriptSig
tx += struct.pack("<I", 0xffffffff)
tx += ser_compact_size(1)
tx += struct.pack("<q", genesisReward)
tx += ser_compact_size(len(scriptPubKey))
tx += scriptPubKey
tx += struct.pack("<I", 0)
merkle = sha256d(tx)
def bits_to_target(bits):
exponent = bits >> 24
mantissa = bits & 0xffffff
return mantissa * (1 << (8 * (exponent - 3)))
target = bits_to_target(nBits)
print("timestamp:", pszTimestamp.decode())
print("time:", nTime)
print("bits:", hex(nBits))
print("merkle:", merkle[::-1].hex())
print("target:", hex(target))
nonce = 0
start = time.time()
while nonce <= 0xffffffff:
header = b""
header += struct.pack("<i", nVersion)
header += b"\x00" * 32
header += merkle
header += struct.pack("<I", nTime)
header += struct.pack("<I", nBits)
header += struct.pack("<I", nonce)
h = sha256d(header)
value = int.from_bytes(h[::-1], "big")
if value <= target:
print()
print("FOUND")
print("nTime =", nTime)
print("nNonce =", nonce)
print("nBits =", hex(nBits))
print("hashGenesisBlock =", h[::-1].hex())
print("hashMerkleRoot =", merkle[::-1].hex())
print("seconds =", round(time.time() - start, 2))
break
if nonce % 1000000 == 0 and nonce > 0:
print("nonce", nonce, "elapsed", round(time.time() - start, 1))
nonce += 1
else:
print("not found")

263
contrib/gitian-build.py Executable file
View file

@ -0,0 +1,263 @@
#!/usr/bin/env python3
# Copyright (c) 2018-2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
import argparse
import os
import subprocess
import sys
def setup():
global args, workdir
programs = ['ruby', 'git', 'make', 'wget', 'curl']
if args.kvm:
programs += ['apt-cacher-ng', 'python-vm-builder', 'qemu-kvm', 'qemu-utils']
elif args.docker:
if not os.path.isfile('/lib/systemd/system/docker.service'):
dockers = ['docker.io', 'docker-ce']
for i in dockers:
return_code = subprocess.call(['sudo', 'apt-get', 'install', '-qq', i])
if return_code == 0:
break
if return_code != 0:
print('Cannot find any way to install Docker.', file=sys.stderr)
sys.exit(1)
else:
programs += ['apt-cacher-ng', 'lxc', 'debootstrap']
subprocess.check_call(['sudo', 'apt-get', 'install', '-qq'] + programs)
if not os.path.isdir('gitian.sigs'):
subprocess.check_call(['git', 'clone', 'https://github.com/digibyte-core/gitian.sigs.git'])
if not os.path.isdir('digibyte-detached-sigs'):
subprocess.check_call(['git', 'clone', 'https://github.com/digibyte-core/digibyte-detached-sigs.git'])
if not os.path.isdir('gitian-builder'):
subprocess.check_call(['git', 'clone', 'https://github.com/devrandom/gitian-builder.git'])
if not os.path.isdir('digibyte'):
subprocess.check_call(['git', 'clone', 'https://github.com/digibyte-core/digibyte.git'])
os.chdir('gitian-builder')
make_image_prog = ['bin/make-base-vm', '--suite', 'focal', '--arch', 'amd64']
if args.docker:
make_image_prog += ['--docker']
elif not args.kvm:
make_image_prog += ['--lxc', '--disksize', '13000']
subprocess.check_call(make_image_prog)
os.chdir(workdir)
if args.is_focal and not args.kvm and not args.docker:
subprocess.check_call(['sudo', 'sed', '-i', 's/lxcbr0/br0/', '/etc/default/lxc-net'])
print('Reboot is required')
sys.exit(0)
def build():
global args, workdir
os.makedirs('digibyte-binaries/' + args.version, exist_ok=True)
print('\nBuilding Dependencies\n')
os.chdir('gitian-builder')
os.makedirs('inputs', exist_ok=True)
subprocess.check_call(['wget', '-O', 'inputs/osslsigncode-2.0.tar.gz', 'https://github.com/mtrojnar/osslsigncode/archive/2.0.tar.gz'])
subprocess.check_call(["echo '5a60e0a4b3e0b4d655317b2f12a810211c50242138322b16e7e01c6fbb89d92f inputs/osslsigncode-2.0.tar.gz' | sha256sum -c"], shell=True)
subprocess.check_call(['make', '-C', '../digibyte/depends', 'download', 'SOURCES_PATH=' + os.getcwd() + '/cache/common'])
if args.linux:
print('\nCompiling ' + args.version + ' Linux')
subprocess.check_call(['bin/gbuild', '-j', args.jobs, '-m', args.memory, '--commit', 'digibyte='+args.commit, '--url', 'digibyte='+args.url, '../digibyte/contrib/gitian-descriptors/gitian-linux.yml'])
subprocess.check_call(['bin/gsign', '-p', args.sign_prog, '--signer', args.signer, '--release', args.version+'-linux', '--destination', '../gitian.sigs/', '../digibyte/contrib/gitian-descriptors/gitian-linux.yml'])
subprocess.check_call('mv build/out/digibyte-*.tar.gz build/out/src/digibyte-*.tar.gz ../digibyte-binaries/'+args.version, shell=True)
if args.windows:
print('\nCompiling ' + args.version + ' Windows')
subprocess.check_call(['bin/gbuild', '-j', args.jobs, '-m', args.memory, '--commit', 'digibyte='+args.commit, '--url', 'digibyte='+args.url, '../digibyte/contrib/gitian-descriptors/gitian-win.yml'])
subprocess.check_call(['bin/gsign', '-p', args.sign_prog, '--signer', args.signer, '--release', args.version+'-win-unsigned', '--destination', '../gitian.sigs/', '../digibyte/contrib/gitian-descriptors/gitian-win.yml'])
subprocess.check_call('mv build/out/digibyte-*-win-unsigned.tar.gz inputs/', shell=True)
subprocess.check_call('mv build/out/digibyte-*.zip build/out/digibyte-*.exe build/out/src/digibyte-*.tar.gz ../digibyte-binaries/'+args.version, shell=True)
if args.macos:
print('\nCompiling ' + args.version + ' MacOS')
subprocess.check_call(['bin/gbuild', '-j', args.jobs, '-m', args.memory, '--commit', 'digibyte='+args.commit, '--url', 'digibyte='+args.url, '../digibyte/contrib/gitian-descriptors/gitian-osx.yml'])
subprocess.check_call(['bin/gsign', '-p', args.sign_prog, '--signer', args.signer, '--release', args.version+'-osx-unsigned', '--destination', '../gitian.sigs/', '../digibyte/contrib/gitian-descriptors/gitian-osx.yml'])
subprocess.check_call('mv build/out/digibyte-*-osx-unsigned.tar.gz inputs/', shell=True)
subprocess.check_call('mv build/out/digibyte-*.tar.gz build/out/digibyte-*.dmg build/out/src/digibyte-*.tar.gz ../digibyte-binaries/'+args.version, shell=True)
os.chdir(workdir)
if args.commit_files:
print('\nCommitting '+args.version+' Unsigned Sigs\n')
os.chdir('gitian.sigs')
subprocess.check_call(['git', 'add', args.version+'-linux/'+args.signer])
subprocess.check_call(['git', 'add', args.version+'-win-unsigned/'+args.signer])
subprocess.check_call(['git', 'add', args.version+'-osx-unsigned/'+args.signer])
subprocess.check_call(['git', 'commit', '-m', 'Add '+args.version+' unsigned sigs for '+args.signer])
os.chdir(workdir)
def sign():
global args, workdir
os.chdir('gitian-builder')
if args.windows:
print('\nSigning ' + args.version + ' Windows')
subprocess.check_call('cp inputs/digibyte-' + args.version + '-win-unsigned.tar.gz inputs/digibyte-win-unsigned.tar.gz', shell=True)
subprocess.check_call(['bin/gbuild', '--skip-image', '--upgrade', '--commit', 'signature='+args.commit, '../digibyte/contrib/gitian-descriptors/gitian-win-signer.yml'])
subprocess.check_call(['bin/gsign', '-p', args.sign_prog, '--signer', args.signer, '--release', args.version+'-win-signed', '--destination', '../gitian.sigs/', '../digibyte/contrib/gitian-descriptors/gitian-win-signer.yml'])
subprocess.check_call('mv build/out/digibyte-*win64-setup.exe ../digibyte-binaries/'+args.version, shell=True)
if args.macos:
print('\nSigning ' + args.version + ' MacOS')
subprocess.check_call('cp inputs/digibyte-' + args.version + '-osx-unsigned.tar.gz inputs/digibyte-osx-unsigned.tar.gz', shell=True)
subprocess.check_call(['bin/gbuild', '--skip-image', '--upgrade', '--commit', 'signature='+args.commit, '../digibyte/contrib/gitian-descriptors/gitian-osx-signer.yml'])
subprocess.check_call(['bin/gsign', '-p', args.sign_prog, '--signer', args.signer, '--release', args.version+'-osx-signed', '--destination', '../gitian.sigs/', '../digibyte/contrib/gitian-descriptors/gitian-osx-signer.yml'])
subprocess.check_call('mv build/out/digibyte-osx-signed.dmg ../digibyte-binaries/'+args.version+'/digibyte-'+args.version+'-osx.dmg', shell=True)
os.chdir(workdir)
if args.commit_files:
print('\nCommitting '+args.version+' Signed Sigs\n')
os.chdir('gitian.sigs')
subprocess.check_call(['git', 'add', args.version+'-win-signed/'+args.signer])
subprocess.check_call(['git', 'add', args.version+'-osx-signed/'+args.signer])
subprocess.check_call(['git', 'commit', '-a', '-m', 'Add '+args.version+' signed binary sigs for '+args.signer])
os.chdir(workdir)
def verify():
global args, workdir
rc = 0
os.chdir('gitian-builder')
print('\nVerifying v'+args.version+' Linux\n')
if subprocess.call(['bin/gverify', '-v', '-d', '../gitian.sigs/', '-r', args.version+'-linux', '../digibyte/contrib/gitian-descriptors/gitian-linux.yml']):
print('Verifying v'+args.version+' Linux FAILED\n')
rc = 1
print('\nVerifying v'+args.version+' Windows\n')
if subprocess.call(['bin/gverify', '-v', '-d', '../gitian.sigs/', '-r', args.version+'-win-unsigned', '../digibyte/contrib/gitian-descriptors/gitian-win.yml']):
print('Verifying v'+args.version+' Windows FAILED\n')
rc = 1
print('\nVerifying v'+args.version+' MacOS\n')
if subprocess.call(['bin/gverify', '-v', '-d', '../gitian.sigs/', '-r', args.version+'-osx-unsigned', '../digibyte/contrib/gitian-descriptors/gitian-osx.yml']):
print('Verifying v'+args.version+' MacOS FAILED\n')
rc = 1
print('\nVerifying v'+args.version+' Signed Windows\n')
if subprocess.call(['bin/gverify', '-v', '-d', '../gitian.sigs/', '-r', args.version+'-win-signed', '../digibyte/contrib/gitian-descriptors/gitian-win-signer.yml']):
print('Verifying v'+args.version+' Signed Windows FAILED\n')
rc = 1
print('\nVerifying v'+args.version+' Signed MacOS\n')
if subprocess.call(['bin/gverify', '-v', '-d', '../gitian.sigs/', '-r', args.version+'-osx-signed', '../digibyte/contrib/gitian-descriptors/gitian-osx-signer.yml']):
print('Verifying v'+args.version+' Signed MacOS FAILED\n')
rc = 1
os.chdir(workdir)
return rc
def main():
global args, workdir
parser = argparse.ArgumentParser(description='Script for running full Gitian builds.')
parser.add_argument('-c', '--commit', action='store_true', dest='commit', help='Indicate that the version argument is for a commit or branch')
parser.add_argument('-p', '--pull', action='store_true', dest='pull', help='Indicate that the version argument is the number of a github repository pull request')
parser.add_argument('-u', '--url', dest='url', default='https://github.com/digibyte/digibyte', help='Specify the URL of the repository. Default is %(default)s')
parser.add_argument('-v', '--verify', action='store_true', dest='verify', help='Verify the Gitian build')
parser.add_argument('-b', '--build', action='store_true', dest='build', help='Do a Gitian build')
parser.add_argument('-s', '--sign', action='store_true', dest='sign', help='Make signed binaries for Windows and MacOS')
parser.add_argument('-B', '--buildsign', action='store_true', dest='buildsign', help='Build both signed and unsigned binaries')
parser.add_argument('-o', '--os', dest='os', default='lwm', help='Specify which Operating Systems the build is for. Default is %(default)s. l for Linux, w for Windows, m for MacOS')
parser.add_argument('-j', '--jobs', dest='jobs', default='2', help='Number of processes to use. Default %(default)s')
parser.add_argument('-m', '--memory', dest='memory', default='2000', help='Memory to allocate in MiB. Default %(default)s')
parser.add_argument('-k', '--kvm', action='store_true', dest='kvm', help='Use KVM instead of LXC')
parser.add_argument('-d', '--docker', action='store_true', dest='docker', help='Use Docker instead of LXC')
parser.add_argument('-S', '--setup', action='store_true', dest='setup', help='Set up the Gitian building environment. Only works on Debian-based systems (Ubuntu, Debian)')
parser.add_argument('-D', '--detach-sign', action='store_true', dest='detach_sign', help='Create the assert file for detached signing. Will not commit anything.')
parser.add_argument('-n', '--no-commit', action='store_false', dest='commit_files', help='Do not commit anything to git')
parser.add_argument('signer', nargs='?', help='GPG signer to sign each build assert file')
parser.add_argument('version', nargs='?', help='Version number, commit, or branch to build. If building a commit or branch, the -c option must be specified')
args = parser.parse_args()
workdir = os.getcwd()
args.is_focal = b'focal' in subprocess.check_output(['lsb_release', '-cs'])
if args.kvm and args.docker:
raise Exception('Error: cannot have both kvm and docker')
# Ensure no more than one environment variable for gitian-builder (USE_LXC, USE_VBOX, USE_DOCKER) is set as they
# can interfere (e.g., USE_LXC being set shadows USE_DOCKER; for details see gitian-builder/libexec/make-clean-vm).
os.environ['USE_LXC'] = ''
os.environ['USE_VBOX'] = ''
os.environ['USE_DOCKER'] = ''
if args.docker:
os.environ['USE_DOCKER'] = '1'
elif not args.kvm:
os.environ['USE_LXC'] = '1'
if 'GITIAN_HOST_IP' not in os.environ.keys():
os.environ['GITIAN_HOST_IP'] = '10.0.3.1'
if 'LXC_GUEST_IP' not in os.environ.keys():
os.environ['LXC_GUEST_IP'] = '10.0.3.5'
if args.setup:
setup()
if args.buildsign:
args.build = True
args.sign = True
if not args.build and not args.sign and not args.verify:
sys.exit(0)
args.linux = 'l' in args.os
args.windows = 'w' in args.os
args.macos = 'm' in args.os
# Disable for MacOS if no SDK found
if args.macos and not os.path.isfile('gitian-builder/inputs/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers.tar.gz'):
print('Cannot build for MacOS, SDK does not exist. Will build for other OSes')
args.macos = False
args.sign_prog = 'true' if args.detach_sign else 'gpg --detach-sign'
script_name = os.path.basename(sys.argv[0])
if not args.signer:
print(script_name+': Missing signer')
print('Try '+script_name+' --help for more information')
sys.exit(1)
if not args.version:
print(script_name+': Missing version')
print('Try '+script_name+' --help for more information')
sys.exit(1)
# Add leading 'v' for tags
if args.commit and args.pull:
raise Exception('Cannot have both commit and pull')
args.commit = ('' if args.commit else 'v') + args.version
os.chdir('digibyte')
if args.pull:
subprocess.check_call(['git', 'fetch', args.url, 'refs/pull/'+args.version+'/merge'])
os.chdir('../gitian-builder/inputs/digibyte')
subprocess.check_call(['git', 'fetch', args.url, 'refs/pull/'+args.version+'/merge'])
args.commit = subprocess.check_output(['git', 'show', '-s', '--format=%H', 'FETCH_HEAD'], universal_newlines=True, encoding='utf8').strip()
args.version = 'pull-' + args.version
print(args.commit)
subprocess.check_call(['git', 'fetch'])
subprocess.check_call(['git', 'checkout', args.commit])
os.chdir(workdir)
os.chdir('gitian-builder')
subprocess.check_call(['git', 'pull'])
os.chdir(workdir)
if args.build:
build()
if args.sign:
sign()
if args.verify:
os.chdir('gitian.sigs')
subprocess.check_call(['git', 'pull'])
os.chdir(workdir)
sys.exit(verify())
if __name__ == '__main__':
main()

387
contrib/gitian-build.sh Executable file
View file

@ -0,0 +1,387 @@
# Copyright (c) 2016 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
# What to do
sign=false
verify=false
build=false
# Systems to build
linux=true
windows=true
osx=true
# Other Basic variables
SIGNER=
VERSION=
commit=false
url=https://github.com/digibyte-core/digibyte
proc=2
mem=2000
lxc=true
osslTarUrl=https://github.com/DigiByte-Core/osslsigncode-1.7.1.tar.gz-1/raw/master/osslsigncode-1.7.1.tar.gz
osslPatchUrl=https://raw.githubusercontent.com/DigiByte-Core/osslsigncode-1.7.1.tar.gz-1/master/osslsigncode-Backports-to-1.7.1.patch
scriptName=$(basename -- "$0")
signProg="gpg --detach-sign"
commitFiles=true
# Help Message
read -d '' usage <<- EOF
Usage: $scriptName [-c|u|v|b|s|B|o|h|j|m|] signer version
Run this script from the directory containing the digibyte, gitian-builder, gitian.sigs, and digibyte-detached-sigs.
Arguments:
signer GPG signer to sign each build assert file
version Version number, commit, or branch to build. If building a commit or branch, the -c option must be specified
Options:
-c|--commit Indicate that the version argument is for a commit or branch
-u|--url Specify the URL of the repository. Default is https://github.com/digibyte-core/digibyte
-v|--verify Verify the Gitian build
-b|--build Do a Gitian build
-s|--sign Make signed binaries for Windows and Mac OSX
-B|--buildsign Build both signed and unsigned binaries
-o|--os Specify which Operating Systems the build is for. Default is lwx. l for linux, w for windows, x for osx
-j Number of processes to use. Default 2
-m Memory to allocate in MiB. Default 2000
--kvm Use KVM instead of LXC
--setup Set up the Gitian building environment. Uses LXC. If you want to use KVM, use the --kvm option. Only works on Debian-based systems (Ubuntu, Debian)
--detach-sign Create the assert file for detached signing. Will not commit anything.
--no-commit Do not commit anything to git
-h|--help Print this help message
EOF
# Get options and arguments
while :; do
case $1 in
# Verify
-v|--verify)
verify=true
;;
# Build
-b|--build)
build=true
;;
# Sign binaries
-s|--sign)
sign=true
;;
# Build then Sign
-B|--buildsign)
sign=true
build=true
;;
# PGP Signer
-S|--signer)
if [ -n "$2" ]
then
SIGNER="$2"
shift
else
echo 'Error: "--signer" requires a non-empty argument.'
exit 1
fi
;;
# Operating Systems
-o|--os)
if [ -n "$2" ]
then
linux=false
windows=false
osx=false
if [[ "$2" = *"l"* ]]
then
linux=true
fi
if [[ "$2" = *"w"* ]]
then
windows=true
fi
if [[ "$2" = *"x"* ]]
then
osx=true
fi
shift
else
echo 'Error: "--os" requires an argument containing an l (for linux), w (for windows), or x (for Mac OSX)'
exit 1
fi
;;
# Help message
-h|--help)
echo "$usage"
exit 0
;;
# Commit or branch
-c|--commit)
commit=true
;;
# Number of Processes
-j)
if [ -n "$2" ]
then
proc=$2
shift
else
echo 'Error: "-j" requires an argument'
exit 1
fi
;;
# Memory to allocate
-m)
if [ -n "$2" ]
then
mem=$2
shift
else
echo 'Error: "-m" requires an argument'
exit 1
fi
;;
# URL
-u)
if [ -n "$2" ]
then
url=$2
shift
else
echo 'Error: "-u" requires an argument'
exit 1
fi
;;
# kvm
--kvm)
lxc=false
;;
# Detach sign
--detach-sign)
signProg="true"
commitFiles=false
;;
# Commit files
--no-commit)
commitFiles=false
;;
# Setup
--setup)
setup=true
;;
*) # Default case: If no more options then break out of the loop.
break
esac
shift
done
# Set up LXC
if [[ $lxc = true ]]
then
export USE_LXC=1
fi
# Check for OSX SDK
if [[ ! -e "gitian-builder/inputs/MacOSX10.11.sdk.tar.gz" && $osx == true ]]
then
echo "Cannot build for OSX, SDK does not exist. Will build for other OSes"
osx=false
fi
# Get signer
if [[ -n "$1" ]]
then
SIGNER="$1"
shift
fi
# Get version
if [[ -n "$1" ]]
then
VERSION=$1
COMMIT=$VERSION
shift
fi
# Check that a signer is specified
if [[ "$SIGNER" == "" ]]
then
echo "$scriptName: Missing signer."
echo "Try $scriptName --help for more information"
exit 1
fi
# Check that a version is specified
if [[ $VERSION == "" ]]
then
echo "$scriptName: Missing version."
echo "Try $scriptName --help for more information"
exit 1
fi
# Add a "v" if no -c
if [[ $commit = false ]]
then
COMMIT="v${VERSION}"
fi
echo ${COMMIT}
# Setup build environment
if [[ $setup = true ]]
then
sudo apt-get install ruby apache2 git apt-cacher-ng python-vm-builder qemu-kvm qemu-utils
git clone https://github.com/digibyte-core/gitian.sigs.git
git clone https://github.com/digibyte-core/digibyte-detached-sigs.git
git clone https://github.com/devrandom/gitian-builder.git
pushd ./gitian-builder
if [[ -n "$USE_LXC" ]]
then
sudo apt-get install lxc
bin/make-base-vm --suite trusty --arch amd64 --lxc
else
bin/make-base-vm --suite trusty --arch amd64
fi
popd
fi
# Set up build
pushd ./digibyte
git fetch
git checkout ${COMMIT}
popd
# Build
if [[ $build = true ]]
then
# Make output folder
mkdir -p ./digibyte-binaries/${VERSION}
# Build Dependencies
echo ""
echo "Building Dependencies"
echo ""
pushd ./gitian-builder
mkdir -p inputs
wget -N -P inputs $osslPatchUrl
wget -N -P inputs $osslTarUrl
make -C ../digibyte/depends download SOURCES_PATH=`pwd`/cache/common
# Linux
if [[ $linux = true ]]
then
echo ""
echo "Compiling ${VERSION} Linux"
echo ""
./bin/gbuild -j ${proc} -m ${mem} --commit digibyte=${COMMIT} --url digibyte=${url} ../digibyte/contrib/gitian-descriptors/gitian-linux.yml
./bin/gsign -p "$signProg" --signer $SIGNER --release ${VERSION}-linux --destination ../gitian.sigs/ ../digibyte/contrib/gitian-descriptors/gitian-linux.yml
mv build/out/digibyte-*.tar.gz build/out/src/digibyte-*.tar.gz ../digibyte-binaries/${VERSION}
fi
# Windows
if [[ $windows = true ]]
then
echo ""
echo "Compiling ${VERSION} Windows"
echo ""
./bin/gbuild -j ${proc} -m ${mem} --commit digibyte=${COMMIT} --url digibyte=${url} ../digibyte/contrib/gitian-descriptors/gitian-win.yml
./bin/gsign -p "$signProg" --signer $SIGNER --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../digibyte/contrib/gitian-descriptors/gitian-win.yml
mv build/out/digibyte-*-win-unsigned.tar.gz inputs/digibyte-win-unsigned.tar.gz
mv build/out/digibyte-*.zip build/out/digibyte-*.exe ../digibyte-binaries/${VERSION}
fi
# Mac OSX
if [[ $osx = true ]]
then
echo ""
echo "Compiling ${VERSION} Mac OSX"
echo ""
./bin/gbuild -j ${proc} -m ${mem} --commit digibyte=${COMMIT} --url digibyte=${url} ../digibyte/contrib/gitian-descriptors/gitian-osx.yml
./bin/gsign -p "$signProg" --signer $SIGNER --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../digibyte/contrib/gitian-descriptors/gitian-osx.yml
mv build/out/digibyte-*-osx-unsigned.tar.gz inputs/digibyte-osx-unsigned.tar.gz
mv build/out/digibyte-*.tar.gz build/out/digibyte-*.dmg ../digibyte-binaries/${VERSION}
fi
popd
if [[ $commitFiles = true ]]
then
# Commit to gitian.sigs repo
echo ""
echo "Committing ${VERSION} Unsigned Sigs"
echo ""
pushd gitian.sigs
git add ${VERSION}-linux/"${SIGNER}"
git add ${VERSION}-win-unsigned/"${SIGNER}"
git add ${VERSION}-osx-unsigned/"${SIGNER}"
git commit -a -m "Add ${VERSION} unsigned sigs for ${SIGNER}"
popd
fi
fi
# Verify the build
if [[ $verify = true ]]
then
# Linux
pushd ./gitian-builder
echo ""
echo "Verifying v${VERSION} Linux"
echo ""
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../digibyte/contrib/gitian-descriptors/gitian-linux.yml
# Windows
echo ""
echo "Verifying v${VERSION} Windows"
echo ""
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../digibyte/contrib/gitian-descriptors/gitian-win.yml
# Mac OSX
echo ""
echo "Verifying v${VERSION} Mac OSX"
echo ""
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../digibyte/contrib/gitian-descriptors/gitian-osx.yml
# Signed Windows
echo ""
echo "Verifying v${VERSION} Signed Windows"
echo ""
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../digibyte/contrib/gitian-descriptors/gitian-osx-signer.yml
# Signed Mac OSX
echo ""
echo "Verifying v${VERSION} Signed Mac OSX"
echo ""
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../digibyte/contrib/gitian-descriptors/gitian-osx-signer.yml
popd
fi
# Sign binaries
if [[ $sign = true ]]
then
pushd ./gitian-builder
# Sign Windows
if [[ $windows = true ]]
then
echo ""
echo "Signing ${VERSION} Windows"
echo ""
./bin/gbuild -i --commit signature=${COMMIT} ../digibyte/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gsign -p "$signProg" --signer "$SIGNER" --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../digibyte/contrib/gitian-descriptors/gitian-win-signer.yml
mv build/out/digibyte-*win64-setup.exe ../digibyte-binaries/${VERSION}
mv build/out/digibyte-*win32-setup.exe ../digibyte-binaries/${VERSION}
fi
# Sign Mac OSX
if [[ $osx = true ]]
then
echo ""
echo "Signing ${VERSION} Mac OSX"
echo ""
./bin/gbuild -i --commit signature=${COMMIT} ../digibyte/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign -p "$signProg" --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../digibyte/contrib/gitian-descriptors/gitian-osx-signer.yml
fi
popd
if [[ $commitFiles = true ]]
then
# Commit Sigs
pushd gitian.sigs
echo ""
echo "Committing ${VERSION} Signed Sigs"
echo ""
git add ${VERSION}-win-signed/"${SIGNER}"
git add ${VERSION}-osx-signed/"${SIGNER}"
git commit -a -m "Add ${VERSION} signed binary sigs for ${SIGNER}"
popd
fi
fi

65
contrib/gitian-descriptors/README.md Normal file
View file

@ -0,0 +1,65 @@
### Gavin's notes on getting gitian builds up and running using KVM
These instructions distilled from
[https://help.ubuntu.com/community/KVM/Installation](https://help.ubuntu.com/community/KVM/Installation).
You need the right hardware: you need a 64-bit-capable CPU with hardware virtualization support (Intel VT-x or AMD-V). Not all modern CPUs support hardware virtualization.
You probably need to enable hardware virtualization in your machine's BIOS.
You need to be running a recent version of 64-bit-Ubuntu, and you need to install several prerequisites:
sudo apt-get install ruby apache2 git apt-cacher-ng python-vm-builder qemu-kvm
Sanity checks:
sudo service apt-cacher-ng status # Should return apt-cacher-ng is running
ls -l /dev/kvm # Should show a /dev/kvm device
Once you've got the right hardware and software:
git clone git://github.com/digibyte/digibyte.git
git clone git://github.com/devrandom/gitian-builder.git
mkdir gitian-builder/inputs
cd gitian-builder/inputs
# Create base images
cd gitian-builder
bin/make-base-vm --suite trusty --arch amd64
cd ..
# Get inputs (see doc/release-process.md for exact inputs needed and where to get them)
...
# For further build instructions see doc/release-process.md
...
---------------------
`gitian-builder` now also supports building using LXC. See
[help.ubuntu.com](https://help.ubuntu.com/14.04/serverguide/lxc.html)
for how to get LXC up and running under Ubuntu.
If your main machine is a 64-bit Mac or PC with a few gigabytes of memory
and at least 10 gigabytes of free disk space, you can `gitian-build` using
LXC running inside a virtual machine.
Here's a description of Gavin's setup on OSX 10.6:
1. Download and install VirtualBox from [https://www.virtualbox.org/](https://www.virtualbox.org/)
2. Download the 64-bit Ubuntu Desktop 12.04 LTS .iso CD image from
[http://www.ubuntu.com/](http://www.ubuntu.com/)
3. Run VirtualBox and create a new virtual machine, using the Ubuntu .iso (see the [VirtualBox documentation](https://www.virtualbox.org/wiki/Documentation) for details). Create it with at least 2 gigabytes of memory and a disk that is at least 20 gigabytes big.
4. Inside the running Ubuntu desktop, install:
sudo apt-get install debootstrap lxc ruby apache2 git apt-cacher-ng python-vm-builder
5. Still inside Ubuntu, tell gitian-builder to use LXC, then follow the "Once you've got the right hardware and software" instructions above:
export USE_LXC=1
git clone git://github.com/digibyte/digibyte.git
... etc

12
contrib/gitian-descriptors/assign_DISTNAME Normal file
View file

@ -0,0 +1,12 @@
# Copyright (c) 2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# A helper script to be sourced into the gitian descriptors
if RECENT_TAG="$(git describe --exact-match HEAD 2> /dev/null)"; then
VERSION="${RECENT_TAG#v}"
else
VERSION="$(git rev-parse --short=12 HEAD)"
fi
DISTNAME="digibyte-${VERSION}"

167
contrib/gitian-descriptors/gitian-linux.yml Normal file
View file

@ -0,0 +1,167 @@
---
name: "digibyte-core-linux-8.22"
enable_cache: true
distro: "ubuntu"
suites:
- "focal"
architectures:
- "amd64"
packages:
# Common dependencies.
- "autoconf"
- "automake"
- "binutils"
- "bison"
- "bsdmainutils"
- "ca-certificates"
- "curl"
- "faketime"
- "g++-8"
- "gcc-8"
- "git"
- "libtool"
- "patch"
- "pkg-config"
- "python3"
- "python3-pip"
# Cross compilation HOSTS:
# - arm-linux-gnueabihf
- "binutils-arm-linux-gnueabihf"
- "g++-8-arm-linux-gnueabihf"
# - aarch64-linux-gnu
- "binutils-aarch64-linux-gnu"
- "g++-8-aarch64-linux-gnu"
# - powerpc64-linux-gnu
- "binutils-powerpc64-linux-gnu"
- "g++-8-powerpc64-linux-gnu"
# - powerpc64le-linux-gnu
- "binutils-powerpc64le-linux-gnu"
- "g++-8-powerpc64le-linux-gnu"
# - riscv64-linux-gnu
- "binutils-riscv64-linux-gnu"
- "g++-8-riscv64-linux-gnu"
remotes:
- "url": "https://github.com/digibyte/digibyte.git"
"dir": "digibyte"
files: []
script: |
set -e -o pipefail
WRAP_DIR=$HOME/wrapped
HOSTS="x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu riscv64-linux-gnu"
CONFIGFLAGS="--enable-glibc-back-compat --enable-reduce-exports --disable-bench --disable-gui-tests --disable-fuzz-binary"
FAKETIME_HOST_PROGS="gcc g++"
FAKETIME_PROGS="date ar ranlib nm"
HOST_CFLAGS="-O2 -g"
HOST_CXXFLAGS="-O2 -g"
HOST_LDFLAGS_BASE="-static-libstdc++ -Wl,-O2"
export TZ="UTC"
export BUILD_DIR="$PWD"
mkdir -p ${WRAP_DIR}
if test -n "$GBUILD_CACHE_ENABLED"; then
export SOURCES_PATH=${GBUILD_COMMON_CACHE}
export BASE_CACHE=${GBUILD_PACKAGE_CACHE}
mkdir -p ${BASE_CACHE} ${SOURCES_PATH}
fi
# Use $LIB in LD_PRELOAD to avoid hardcoding the dir (See `man ld.so`)
function create_global_faketime_wrappers {
for prog in ${FAKETIME_PROGS}; do
echo '#!/usr/bin/env bash' > ${WRAP_DIR}/${prog}
echo "REAL=\`which -a ${prog} | grep -v ${WRAP_DIR}/${prog} | head -1\`" >> ${WRAP_DIR}/${prog}
echo "export LD_PRELOAD='/usr/\$LIB/faketime/libfaketime.so.1'" >> ${WRAP_DIR}/${prog}
echo "export FAKETIME=\"$1\"" >> ${WRAP_DIR}/${prog}
echo "exec \"\$REAL\" \"\$@\"" >> $WRAP_DIR/${prog}
chmod +x ${WRAP_DIR}/${prog}
done
}
function create_per-host_faketime_wrappers {
for i in $HOSTS; do
for prog in ${FAKETIME_HOST_PROGS}; do
if which ${i}-${prog}-8
then
echo '#!/usr/bin/env bash' > ${WRAP_DIR}/${i}-${prog}
echo "REAL=\`which -a ${i}-${prog}-8 | grep -v ${WRAP_DIR}/${i}-${prog} | head -1\`" >> ${WRAP_DIR}/${i}-${prog}
echo "export LD_PRELOAD='/usr/\$LIB/faketime/libfaketime.so.1'" >> ${WRAP_DIR}/${i}-${prog}
echo "export FAKETIME=\"$1\"" >> ${WRAP_DIR}/${i}-${prog}
if [ "${i:0:11}" = "powerpc64le" ]; then
echo "exec \"\$REAL\" -mcpu=power8 -mtune=power9 \"\$@\"" >> $WRAP_DIR/${i}-${prog}
elif [ "${i:0:9}" = "powerpc64" ]; then
echo "exec \"\$REAL\" -mcpu=970 -mtune=power9 \"\$@\"" >> $WRAP_DIR/${i}-${prog}
else
echo "exec \"\$REAL\" \"\$@\"" >> $WRAP_DIR/${i}-${prog}
fi
chmod +x ${WRAP_DIR}/${i}-${prog}
fi
done
done
}
pip3 install lief==0.11.5
# Faketime for depends so intermediate results are comparable
export PATH_orig=${PATH}
create_global_faketime_wrappers "2000-01-01 12:00:00"
create_per-host_faketime_wrappers "2000-01-01 12:00:00"
export PATH=${WRAP_DIR}:${PATH}
cd digibyte
BASEPREFIX="${PWD}/depends"
# Build dependencies for each host
for i in $HOSTS; do
make ${MAKEOPTS} -C ${BASEPREFIX} HOST="${i}" CC=${i}-gcc-8 CXX=${i}-g++-8
done
# Faketime for binaries
export PATH=${PATH_orig}
create_global_faketime_wrappers "${REFERENCE_DATETIME}"
create_per-host_faketime_wrappers "${REFERENCE_DATETIME}"
export PATH=${WRAP_DIR}:${PATH}
# Define DISTNAME variable.
# shellcheck source=contrib/gitian-descriptors/assign_DISTNAME
source contrib/gitian-descriptors/assign_DISTNAME
GIT_ARCHIVE="${OUTDIR}/src/${DISTNAME}.tar.gz"
# Create the source tarball
mkdir -p "$(dirname "$GIT_ARCHIVE")"
git archive --prefix="${DISTNAME}/" --output="$GIT_ARCHIVE" HEAD
ORIGPATH="$PATH"
# Extract the git archive into a dir for each host and build
for i in ${HOSTS}; do
export PATH=${BASEPREFIX}/${i}/native/bin:${ORIGPATH}
if [ "${i}" = "powerpc64-linux-gnu" ]; then
# Workaround for https://bugs.launchpad.net/ubuntu/+source/gcc-8-cross-ports/+bug/1853740
# TODO: remove this when no longer needed
HOST_LDFLAGS="${HOST_LDFLAGS_BASE} -Wl,-z,noexecstack"
else
HOST_LDFLAGS="${HOST_LDFLAGS_BASE}"
fi
mkdir -p distsrc-${i}
cd distsrc-${i}
INSTALLPATH="${PWD}/installed/${DISTNAME}"
mkdir -p ${INSTALLPATH}
tar --strip-components=1 -xf "${GIT_ARCHIVE}"
./autogen.sh
CONFIG_SITE=${BASEPREFIX}/${i}/share/config.site ./configure --prefix=/ --disable-ccache --disable-maintainer-mode --disable-dependency-tracking ${CONFIGFLAGS} CFLAGS="${HOST_CFLAGS}" CXXFLAGS="${HOST_CXXFLAGS}" LDFLAGS="${HOST_LDFLAGS}" CC=${i}-gcc-8 CXX=${i}-g++-8
make ${MAKEOPTS}
make ${MAKEOPTS} -C src check-security
make ${MAKEOPTS} -C src check-symbols
make install DESTDIR=${INSTALLPATH}
cd installed
find . -name "lib*.la" -delete
find . -name "lib*.a" -delete
rm -rf ${DISTNAME}/lib/pkgconfig
find ${DISTNAME}/bin -type f -executable -print0 | xargs -0 -n1 -I{} ../contrib/devtools/split-debug.sh {} {} {}.dbg
find ${DISTNAME}/lib -type f -print0 | xargs -0 -n1 -I{} ../contrib/devtools/split-debug.sh {} {} {}.dbg
cp ../README.md ${DISTNAME}/
find ${DISTNAME} -not -name "*.dbg" | sort | tar --mtime="$REFERENCE_DATETIME" --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}-${i}.tar.gz
find ${DISTNAME} -name "*.dbg" | sort | tar --mtime="$REFERENCE_DATETIME" --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}-${i}-debug.tar.gz
cd ../../
rm -rf distsrc-${i}
done

53
contrib/gitian-descriptors/gitian-osx-signer.yml Normal file
View file

@ -0,0 +1,53 @@
---
name: "digibyte-dmg-signer"
distro: "ubuntu"
suites:
- "focal"
architectures:
- "amd64"
packages:
- "faketime"
- "xorriso"
- "python3-pip"
remotes:
- "url": "https://github.com/digibyte-core/digibyte-detached-sigs.git"
"dir": "signature"
- "url": "https://github.com/achow101/signapple.git"
"dir": "signapple"
"commit": "b084cbbf44d5330448ffce0c7d118f75781b64bd"
files:
- "digibyte-osx-unsigned.tar.gz"
script: |
set -e -o pipefail
WRAP_DIR=$HOME/wrapped
mkdir -p ${WRAP_DIR}
export PATH="$PWD":$PATH
FAKETIME_PROGS="dmg xorrisofs"
# Create global faketime wrappers
for prog in ${FAKETIME_PROGS}; do
echo '#!/usr/bin/env bash' > ${WRAP_DIR}/${prog}
echo "REAL=\`which -a ${prog} | grep -v ${WRAP_DIR}/${prog} | head -1\`" >> ${WRAP_DIR}/${prog}
echo "export LD_PRELOAD='/usr/\$LIB/faketime/libfaketime.so.1'" >> ${WRAP_DIR}/${prog}
echo "export FAKETIME=\"${REFERENCE_DATETIME}\"" >> ${WRAP_DIR}/${prog}
echo "exec \"\$REAL\" \"\$@\"" >> $WRAP_DIR/${prog}
chmod +x ${WRAP_DIR}/${prog}
done
# Install signapple
cd signapple
python3 -m pip install -U pip setuptools
python3 -m pip install .
export PATH="$HOME/.local/bin":$PATH
cd ..
UNSIGNED_TARBALL=digibyte-osx-unsigned.tar.gz
UNSIGNED_APP=dist/DigiByte-Qt.app
SIGNED=digibyte-osx-signed.dmg
tar -xf ${UNSIGNED_TARBALL}
OSX_VOLNAME="$(cat osx_volname)"
./detached-sig-apply.sh ${UNSIGNED_APP} signature/osx/dist
${WRAP_DIR}/xorrisofs -D -l -V "${OSX_VOLNAME}" -no-pad -r -dir-mode 0755 -o uncompressed.dmg signed-app
${WRAP_DIR}/dmg dmg uncompressed.dmg ${OUTDIR}/${SIGNED}

155
contrib/gitian-descriptors/gitian-osx.yml Normal file
View file

@ -0,0 +1,155 @@
---
name: "digibyte-core-osx-8.22"
enable_cache: true
distro: "ubuntu"
suites:
- "focal"
architectures:
- "amd64"
packages:
- "ca-certificates"
- "curl"
- "g++"
- "git"
- "pkg-config"
- "autoconf"
- "librsvg2-bin"
- "libtiff-tools"
- "libtool"
- "automake"
- "faketime"
- "bsdmainutils"
- "cmake"
- "imagemagick"
- "libz-dev"
- "python3"
- "python3-pip"
- "python3-setuptools"
- "fonts-tuffy"
- "xorriso"
- "libtinfo5"
remotes:
- "url": "https://github.com/digibyte/digibyte.git"
"dir": "digibyte"
files:
- "Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers.tar.gz"
script: |
set -e -o pipefail
WRAP_DIR=$HOME/wrapped
HOSTS="x86_64-apple-darwin18"
CONFIGFLAGS="--enable-reduce-exports --disable-bench --disable-gui-tests --disable-fuzz-binary XORRISOFS=${WRAP_DIR}/xorrisofs DMG=${WRAP_DIR}/dmg"
FAKETIME_HOST_PROGS=""
FAKETIME_PROGS="ar ranlib date dmg xorrisofs"
export TZ="UTC"
export BUILD_DIR="$PWD"
mkdir -p ${WRAP_DIR}
if test -n "$GBUILD_CACHE_ENABLED"; then
export SOURCES_PATH=${GBUILD_COMMON_CACHE}
export BASE_CACHE=${GBUILD_PACKAGE_CACHE}
mkdir -p ${BASE_CACHE} ${SOURCES_PATH}
fi
export ZERO_AR_DATE=1
# Use $LIB in LD_PRELOAD to avoid hardcoding the dir (See `man ld.so`)
function create_global_faketime_wrappers {
for prog in ${FAKETIME_PROGS}; do
echo '#!/usr/bin/env bash' > ${WRAP_DIR}/${prog}
echo "REAL=\`which -a ${prog} | grep -v ${WRAP_DIR}/${prog} | head -1\`" >> ${WRAP_DIR}/${prog}
echo "export LD_PRELOAD='/usr/\$LIB/faketime/libfaketime.so.1'" >> ${WRAP_DIR}/${prog}
echo "export FAKETIME=\"$1\"" >> ${WRAP_DIR}/${prog}
echo "exec \"\$REAL\" \"\$@\"" >> $WRAP_DIR/${prog}
chmod +x ${WRAP_DIR}/${prog}
done
}
function create_per-host_faketime_wrappers {
for i in $HOSTS; do
for prog in ${FAKETIME_HOST_PROGS}; do
echo '#!/usr/bin/env bash' > ${WRAP_DIR}/${i}-${prog}
echo "REAL=\`which -a ${i}-${prog} | grep -v ${WRAP_DIR}/${i}-${prog} | head -1\`" >> ${WRAP_DIR}/${i}-${prog}
echo "export LD_PRELOAD='/usr/\$LIB/faketime/libfaketime.so.1'" >> ${WRAP_DIR}/${i}-${prog}
echo "export FAKETIME=\"$1\"" >> ${WRAP_DIR}/${i}-${prog}
echo "exec \"\$REAL\" \"\$@\"" >> $WRAP_DIR/${i}-${prog}
chmod +x ${WRAP_DIR}/${i}-${prog}
done
done
}
pip3 install lief==0.11.5
# Faketime for depends so intermediate results are comparable
export PATH_orig=${PATH}
create_global_faketime_wrappers "2000-01-01 12:00:00"
create_per-host_faketime_wrappers "2000-01-01 12:00:00"
export PATH=${WRAP_DIR}:${PATH}
cd digibyte
BASEPREFIX="${PWD}/depends"
mkdir -p ${BASEPREFIX}/SDKs
tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers.tar.gz
# Build dependencies for each host
for i in $HOSTS; do
make ${MAKEOPTS} -C ${BASEPREFIX} HOST="${i}"
done
# Faketime for binaries
export PATH=${PATH_orig}
create_global_faketime_wrappers "${REFERENCE_DATETIME}"
create_per-host_faketime_wrappers "${REFERENCE_DATETIME}"
export PATH=${WRAP_DIR}:${PATH}
# Define DISTNAME variable.
# shellcheck source=contrib/gitian-descriptors/assign_DISTNAME
source contrib/gitian-descriptors/assign_DISTNAME
GIT_ARCHIVE="${OUTDIR}/src/${DISTNAME}.tar.gz"
# Create the source tarball
mkdir -p "$(dirname "$GIT_ARCHIVE")"
git archive --prefix="${DISTNAME}/" --output="$GIT_ARCHIVE" HEAD
ORIGPATH="$PATH"
# Extract the git archive into a dir for each host and build
for i in ${HOSTS}; do
export PATH=${BASEPREFIX}/${i}/native/bin:${ORIGPATH}
mkdir -p distsrc-${i}
cd distsrc-${i}
INSTALLPATH="${PWD}/installed/${DISTNAME}"
mkdir -p ${INSTALLPATH}
tar --strip-components=1 -xf "${GIT_ARCHIVE}"
./autogen.sh
CONFIG_SITE=${BASEPREFIX}/${i}/share/config.site ./configure --prefix=/ --disable-ccache --disable-maintainer-mode --disable-dependency-tracking ${CONFIGFLAGS}
make ${MAKEOPTS}
make ${MAKEOPTS} -C src check-security
make ${MAKEOPTS} -C src check-symbols
make install-strip DESTDIR=${INSTALLPATH}
make osx_volname
make deploydir
mkdir -p unsigned-app-${i}
cp osx_volname unsigned-app-${i}/
cp contrib/macdeploy/detached-sig-apply.sh unsigned-app-${i}
cp contrib/macdeploy/detached-sig-create.sh unsigned-app-${i}
cp ${BASEPREFIX}/${i}/native/bin/dmg unsigned-app-${i}
mv dist unsigned-app-${i}
pushd unsigned-app-${i}
find . | sort | tar --mtime="$REFERENCE_DATETIME" --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}-osx-unsigned.tar.gz
popd
make deploy OSX_DMG="${OUTDIR}/${DISTNAME}-osx-unsigned.dmg"
cd installed
find . -name "lib*.la" -delete
find . -name "lib*.a" -delete
rm -rf ${DISTNAME}/lib/pkgconfig
find ${DISTNAME} | sort | tar --mtime="$REFERENCE_DATETIME" --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}-${i}.tar.gz
cd ../../
done
mv ${OUTDIR}/${DISTNAME}-x86_64-*.tar.gz ${OUTDIR}/${DISTNAME}-osx64.tar.gz

42
contrib/gitian-descriptors/gitian-win-signer.yml Normal file
View file

@ -0,0 +1,42 @@
---
name: "digibyte-win-signer"
distro: "ubuntu"
suites:
- "focal"
architectures:
- "amd64"
packages:
- "libssl-dev"
- "autoconf"
- "automake"
- "libtool"
- "pkg-config"
remotes:
- "url": "https://github.com/digibyte-core/digibyte-detached-sigs.git"
"dir": "signature"
files:
- "osslsigncode-2.0.tar.gz"
- "digibyte-win-unsigned.tar.gz"
script: |
set -e -o pipefail
BUILD_DIR="$PWD"
SIGDIR=${BUILD_DIR}/signature/win
UNSIGNED_DIR=${BUILD_DIR}/unsigned
echo "5a60e0a4b3e0b4d655317b2f12a810211c50242138322b16e7e01c6fbb89d92f osslsigncode-2.0.tar.gz" | sha256sum -c
mkdir -p ${UNSIGNED_DIR}
tar -C ${UNSIGNED_DIR} -xf digibyte-win-unsigned.tar.gz
tar xf osslsigncode-2.0.tar.gz
cd osslsigncode-2.0
./autogen.sh
./configure --without-gsf --without-curl --disable-dependency-tracking
make
find ${UNSIGNED_DIR} -name "*-unsigned.exe" | while read i; do
INFILE="$(basename "${i}")"
OUTFILE="${INFILE/-unsigned}"
./osslsigncode attach-signature -in "${i}" -out "${OUTDIR}/${OUTFILE}" -sigin "${SIGDIR}/${INFILE}.pem"
done

157
contrib/gitian-descriptors/gitian-win.yml Normal file
View file

@ -0,0 +1,157 @@
---
name: "digibyte-core-win-8.22"
enable_cache: true
distro: "ubuntu"
suites:
- "focal"
architectures:
- "amd64"
packages:
- "curl"
- "g++"
- "git"
- "pkg-config"
- "autoconf"
- "libtool"
- "automake"
- "faketime"
- "bsdmainutils"
- "mingw-w64"
- "g++-mingw-w64"
- "nsis"
- "zip"
- "ca-certificates"
- "python3"
- "python3-pip"
remotes:
- "url": "https://github.com/digibyte/digibyte.git"
"dir": "digibyte"
files: []
script: |
set -e -o pipefail
WRAP_DIR=$HOME/wrapped
HOSTS="x86_64-w64-mingw32"
CONFIGFLAGS="--enable-reduce-exports --disable-bench --disable-gui-tests --disable-fuzz-binary"
FAKETIME_HOST_PROGS="ar ranlib nm windres strip objcopy"
FAKETIME_PROGS="date makensis zip"
HOST_CFLAGS="-O2 -g -fno-ident"
HOST_CXXFLAGS="-O2 -g -fno-ident"
export TZ="UTC"
export BUILD_DIR="$PWD"
mkdir -p ${WRAP_DIR}
if test -n "$GBUILD_CACHE_ENABLED"; then
export SOURCES_PATH=${GBUILD_COMMON_CACHE}
export BASE_CACHE=${GBUILD_PACKAGE_CACHE}
mkdir -p ${BASE_CACHE} ${SOURCES_PATH}
fi
# Use $LIB in LD_PRELOAD to avoid hardcoding the dir (See `man ld.so`)
function create_global_faketime_wrappers {
for prog in ${FAKETIME_PROGS}; do
echo '#!/usr/bin/env bash' > ${WRAP_DIR}/${prog}
echo "REAL=\`which -a ${prog} | grep -v ${WRAP_DIR}/${prog} | head -1\`" >> ${WRAP_DIR}/${prog}
echo "export LD_PRELOAD='/usr/\$LIB/faketime/libfaketime.so.1'" >> ${WRAP_DIR}/${prog}
echo "export FAKETIME=\"$1\"" >> ${WRAP_DIR}/${prog}
echo "exec \"\$REAL\" \"\$@\"" >> $WRAP_DIR/${prog}
chmod +x ${WRAP_DIR}/${prog}
done
}
function create_per-host_faketime_wrappers {
for i in $HOSTS; do
for prog in ${FAKETIME_HOST_PROGS}; do
echo '#!/usr/bin/env bash' > ${WRAP_DIR}/${i}-${prog}
echo "REAL=\`which -a ${i}-${prog} | grep -v ${WRAP_DIR}/${i}-${prog} | head -1\`" >> ${WRAP_DIR}/${i}-${prog}
echo "export LD_PRELOAD='/usr/\$LIB/faketime/libfaketime.so.1'" >> ${WRAP_DIR}/${i}-${prog}
echo "export FAKETIME=\"$1\"" >> ${WRAP_DIR}/${i}-${prog}
echo "exec \"\$REAL\" \"\$@\"" >> $WRAP_DIR/${i}-${prog}
chmod +x ${WRAP_DIR}/${i}-${prog}
done
done
}
function create_per-host_compiler_wrapper {
# -posix variant is required for c++11 threading.
for i in $HOSTS; do
for prog in gcc g++; do
echo '#!/usr/bin/env bash' > ${WRAP_DIR}/${i}-${prog}
echo "REAL=\`which -a ${i}-${prog}-posix | grep -v ${WRAP_DIR}/${i}-${prog} | head -1\`" >> ${WRAP_DIR}/${i}-${prog}
echo "export LD_PRELOAD='/usr/\$LIB/faketime/libfaketime.so.1'" >> ${WRAP_DIR}/${i}-${prog}
echo "export FAKETIME=\"$1\"" >> ${WRAP_DIR}/${i}-${prog}
echo "exec \"\$REAL\" \"\$@\"" >> $WRAP_DIR/${i}-${prog}
chmod +x ${WRAP_DIR}/${i}-${prog}
done
done
}
pip3 install lief==0.11.5
# Faketime for depends so intermediate results are comparable
export PATH_orig=${PATH}
create_global_faketime_wrappers "2000-01-01 12:00:00"
create_per-host_faketime_wrappers "2000-01-01 12:00:00"
create_per-host_compiler_wrapper "2000-01-01 12:00:00"
export PATH=${WRAP_DIR}:${PATH}
cd digibyte
BASEPREFIX="${PWD}/depends"
# Build dependencies for each host
for i in $HOSTS; do
make ${MAKEOPTS} -C ${BASEPREFIX} HOST="${i}"
done
# Faketime for binaries
export PATH=${PATH_orig}
create_global_faketime_wrappers "${REFERENCE_DATETIME}"
create_per-host_faketime_wrappers "${REFERENCE_DATETIME}"
create_per-host_compiler_wrapper "${REFERENCE_DATETIME}"
export PATH=${WRAP_DIR}:${PATH}
# Define DISTNAME variable.
# shellcheck source=contrib/gitian-descriptors/assign_DISTNAME
source contrib/gitian-descriptors/assign_DISTNAME
GIT_ARCHIVE="${OUTDIR}/src/${DISTNAME}.tar.gz"
# Create the source tarball
mkdir -p "$(dirname "$GIT_ARCHIVE")"
git archive --prefix="${DISTNAME}/" --output="$GIT_ARCHIVE" HEAD
ORIGPATH="$PATH"
# Extract the git archive into a dir for each host and build
for i in ${HOSTS}; do
export PATH=${BASEPREFIX}/${i}/native/bin:${ORIGPATH}
mkdir -p distsrc-${i}
cd distsrc-${i}
INSTALLPATH="${PWD}/installed/${DISTNAME}"
mkdir -p ${INSTALLPATH}
tar --strip-components=1 -xf "${GIT_ARCHIVE}"
./autogen.sh
CONFIG_SITE=${BASEPREFIX}/${i}/share/config.site ./configure --prefix=/ --disable-ccache --disable-maintainer-mode --disable-dependency-tracking ${CONFIGFLAGS} CFLAGS="${HOST_CFLAGS}" CXXFLAGS="${HOST_CXXFLAGS}"
make ${MAKEOPTS}
make ${MAKEOPTS} -C src check-security
make ${MAKEOPTS} -C src check-symbols
make deploy DIGIBYTE_WIN_INSTALLER="${OUTDIR}/${DISTNAME}-win64-setup-unsigned.exe"
make install DESTDIR=${INSTALLPATH}
cd installed
mv ${DISTNAME}/bin/*.dll ${DISTNAME}/lib/
find . -name "lib*.la" -delete
find . -name "lib*.a" -delete
rm -rf ${DISTNAME}/lib/pkgconfig
find ${DISTNAME}/bin -type f -executable -print0 | xargs -0 -n1 -I{} ../contrib/devtools/split-debug.sh {} {} {}.dbg
find ${DISTNAME}/lib -type f -print0 | xargs -0 -n1 -I{} ../contrib/devtools/split-debug.sh {} {} {}.dbg
cp ../doc/README_windows.txt ${DISTNAME}/readme.txt
find ${DISTNAME} -not -name "*.dbg" -type f | sort | zip -X@ ${OUTDIR}/${DISTNAME}-${i//x86_64-w64-mingw32/win64}.zip
find ${DISTNAME} -name "*.dbg" -type f | sort | zip -X@ ${OUTDIR}/${DISTNAME}-${i//x86_64-w64-mingw32/win64}-debug.zip
cd ../../
rm -rf distsrc-${i}
done
cp -rf contrib/windeploy $BUILD_DIR
cd $BUILD_DIR/windeploy
mkdir unsigned
cp ${OUTDIR}/${DISTNAME}-win64-setup-unsigned.exe unsigned/
find . | sort | tar --mtime="$REFERENCE_DATETIME" --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}-win-unsigned.tar.gz

7
contrib/gitian-keys/keys.txt Normal file
View file

@ -0,0 +1,7 @@
63250DC0970025475B4E25FF0939B74825FD3E8C GTO90
9D4FFBD5132363C075D457EF761DCC6FD102ED18 Jared Tate
EAB3D04BA093CC5C361D1458CF9E3A369FB3C44F Gary Mckee
F0A73AD2753B6B4B0EC92EABB4F69792F1502708 Josiah Spackman
BB7C1048F80B0A7BD300806AE01EFDE2E032D813 John Song
F50C7A213871356554FBA7A839C7549323B89CF8 Yoshi Jäger
2C36B291C351ABB9F0668735EDAD99D699AEA9E7 ycagel

813
contrib/guix/INSTALL.md Normal file
View file

@ -0,0 +1,813 @@
# Guix Installation and Setup
This only needs to be done once per machine. If you have already completed the
installation and setup, please proceed to [perform a build](./README.md).
Otherwise, you may choose from one of the following options to install Guix:
1. Using the official **shell installer script** [⤓ skip to section][install-script]
- Maintained by Guix developers
- Easiest (automatically performs *most* setup)
- Works on nearly all Linux distributions
- Only installs latest release
- Binary installation only, requires high level of trust
- Note: The script needs to be run as root, so it should be inspected before it's run
2. Using the official **binary tarball** [⤓ skip to section][install-bin-tarball]
- Maintained by Guix developers
- Normal difficulty (full manual setup required)
- Works on nearly all Linux distributions
- Installs any release
- Binary installation only, requires high level of trust
3. Using fanquake's **Docker image** [↗︎ external instructions][install-fanquake-docker]
- Maintained by fanquake
- Easy (automatically performs *some* setup)
- Works wherever Docker images work
- Installs any release
- Binary installation only, requires high level of trust
4. Using a **distribution-maintained package** [⤓ skip to section][install-distro-pkg]
- Maintained by distribution's Guix package maintainer
- Normal difficulty (manual setup required)
- Works only on distributions with Guix packaged, see: https://repology.org/project/guix/versions
- Installs a release decided on by package maintainer
- Source or binary installation depending on the distribution
5. Building **from source** [⤓ skip to section][install-source]
- Maintained by you
- Hard, but rewarding
- Can be made to work on most Linux distributions
- Installs any commit (more granular)
- Source installation, requires lower level of trust
## Options 1 and 2: Using the official shell installer script or binary tarball
The installation instructions for both the official shell installer script and
the binary tarballs can be found in the GNU Guix Manual's [Binary Installation
section](https://guix.gnu.org/manual/en/html_node/Binary-Installation.html).
Note that running through the binary tarball installation steps is largely
equivalent to manually performing what the shell installer script does.
Note that at the time of writing (July 5th, 2021), the shell installer script
automatically creates an `/etc/profile.d` entry which the binary tarball
installation instructions do not ask you to create. However, you will likely
need this entry for better desktop integration. Please see [this
section](#add-an-etcprofiled-entry) for instructions on how to add a
`/etc/profile.d/guix.sh` entry.
Regardless of which installation option you chose, the changes to
`/etc/profile.d` will not take effect until the next shell or desktop session,
so you should log out and log back in.
## Option 3: Using fanquake's Docker image
Please refer to fanquake's instructions
[here](https://github.com/fanquake/core-review/tree/master/guix).
Note that the `Dockerfile` is largely equivalent to running through the binary
tarball installation steps.
## Option 4: Using a distribution-maintained package
Note that this section is based on the distro packaging situation at the time of
writing (July 2021). Guix is expected to be more widely packaged over time. For
an up-to-date view on Guix's package status/version across distros, please see:
https://repology.org/project/guix/versions
### Debian 11 (Bullseye)/Ubuntu 21.04 (Hirsute Hippo)
Guix v1.2.0 is available as a distribution package starting in [Debian
11](https://packages.debian.org/bullseye/guix) and [Ubuntu
21.04](https://packages.ubuntu.com/hirsute/guix).
Note that if you intend on using Guix without using any substitutes (more
details [here][security-model]), v1.2.0 has a known problems when building
GnuTLS from source. Solutions and workarounds are documented
[here](#gnutls-test-suite-fail-status-request-revoked).
To install:
```sh
sudo apt install guix
```
For up-to-date information on Debian and Ubuntu's release history:
- [Debian release history](https://www.debian.org/releases/)
- [Ubuntu release history](https://ubuntu.com/about/release-cycle)
### Arch Linux
Guix is available in the AUR as
[`guix`](https://aur.archlinux.org/packages/guix/), please follow the
installation instructions in the Arch Linux Wiki ([live
link](https://wiki.archlinux.org/index.php/Guix#AUR_Package_Installation),
[2021/03/30
permalink](https://wiki.archlinux.org/index.php?title=Guix&oldid=637559#AUR_Package_Installation))
to install Guix.
At the time of writing (2021/03/30), the `check` phase will fail if the path to
guix's build directory is longer than 36 characters due to an anachronistic
character limit on the shebang line. Since the `check` phase happens after the
`build` phase, which may take quite a long time, it is recommended that users
either:
1. Skip the `check` phase
- For `makepkg`: `makepkg --nocheck ...`
- For `yay`: `yay --mflags="--nocheck" ...`
- For `paru`: `paru --nocheck ...`
2. Or, check their build directory's length beforehand
- For those building with `makepkg`: `pwd | wc -c`
## Option 5: Building from source
Building Guix from source is a rather involved process but a rewarding one for
those looking to minimize trust and maximize customizability (e.g. building a
particular commit of Guix). Previous experience with using autotools-style build
systems to build packages from source will be helpful. *hic sunt dracones.*
I strongly urge you to at least skim through the entire section once before you
start issuing commands, as it will save you a lot of unncessary pain and
anguish.
### Installing common build tools
There are a few basic build tools that are required for most things we'll build,
so let's install them now:
Text transformation/i18n:
- `autopoint` (sometimes packaged in `gettext`)
- `help2man`
- `po4a`
- `texinfo`
Build system tools:
- `g++` w/ C++11 support
- `libtool`
- `autoconf`
- `automake`
- `pkg-config` (sometimes packaged as `pkgconf`)
- `make`
- `cmake`
Miscellaneous:
- `git`
- `gnupg`
- `python3`
### Building and Installing Guix's dependencies
In order to build Guix itself from source, we need to first make sure that the
necessary dependencies are installed and discoverable. The most up-to-date list
of Guix's dependencies is kept in the ["Requirements"
section](https://guix.gnu.org/manual/en/html_node/Requirements.html) of the Guix
Reference Manual.
Depending on your distribution, most or all of these dependencies may already be
packaged and installable without manually building and installing.
For reference, the graphic below outlines Guix v1.3.0's dependency graph:
![boostrap map](https://user-images.githubusercontent.com/6399679/125064185-a9a59880-e0b0-11eb-82c1-9b8e5dc9950d.png)
#### Guile
##### Choosing a Guile version and sticking to it
One of the first things you need to decide is which Guile version you want to
use: Guile v2.2 or Guile v3.0. Unlike the python2 to python3 transition, Guile
v2.2 and Guile v3.0 are largely compatible, as evidenced by the fact that most
Guile packages and even [Guix
itself](https://guix.gnu.org/en/blog/2020/guile-3-and-guix/) support running on
both.
What is important here is that you **choose one**, and you **remain consistent**
with your choice throughout **all Guile-related packages**, no matter if they
are installed via the distribution's package manager or installed from source.
This is because the files for Guile packages are installed to directories which
are separated based on the Guile version.
###### Example: Checking that Ubuntu's `guile-git` is compatible with your chosen Guile version
On Ubuntu Focal:
```sh
$ apt show guile-git
Package: guile-git
...
Depends: guile-2.2, guile-bytestructures, libgit2-dev
...
```
As you can see, the package `guile-git` depends on `guile-2.2`, meaning that it
was likely built for Guile v2.2. This means that if you decided to use Guile
v3.0 on Ubuntu Focal, you would need to build guile-git from source instead of
using the distribution package.
On Ubuntu Hirsute:
```sh
$ apt show guile-git
Package: guile-git
...
Depends: guile-3.0 | guile-2.2, guile-bytestructures (>= 1.0.7-3~), libgit2-dev (>= 1.0)
...
```
In this case, `guile-git` depends on either `guile-3.0` or `guile-2.2`, meaning
that it would work no matter what Guile version you decided to use.
###### Corner case: Multiple versions of Guile on one system
It is recommended to only install one version of Guile, so that build systems do
not get confused about which Guile to use.
However, if you insist on having both Guile v2.2 and Guile v3.0 installed on
your system, then you need to **consistently** specify one of
`GUILE_EFFECTIVE_VERSION=3.0` or `GUILE_EFFECTIVE_VERSION=2.2` to all
`./configure` invocations for Guix and its dependencies.
##### Installing Guile
Guile is most likely already packaged for your distribution, so after you have
[chosen a Guile version](#choosing-a-guile-version-and-sticking-to-it), install
it via your distribution's package manager.
If your distribution splits packages into `-dev`-suffixed and
non-`-dev`-suffixed sub-packages (as is the case for Debian-derived
distributions), please make sure to install both. For example, to install Guile
v2.2 on Debian/Ubuntu:
```sh
apt install guile-2.2 guile-2.2-dev
```
#### Mixing distribution packages and source-built packages
At the time of writing, most distributions have _some_ of Guix's dependencies
packaged, but not all. This means that you may want to install the distribution
package for some dependencies, and manually build-from-source for others.
Distribution packages usually install to `/usr`, which is different from the
default `./configure` prefix of source-built packages: `/usr/local`.
This means that if you mix-and-match distribution packages and source-built
packages and do not specify exactly `--prefix=/usr` to `./configure` for
source-built packages, you will need to augment the `GUILE_LOAD_PATH` and
`GUILE_LOAD_COMPILED_PATH` environment variables so that Guile will look
under the right prefix and find your source-built packages.
For example, if you are using Guile v2.2, and have Guile packages in the
`/usr/local` prefix, either add the following lines to your `.profile` or
`.bash_profile` so that the environment variable is properly set for all future
shell logins, or paste the lines into a POSIX-style shell to temporarily modify
the environment variables of your current shell session.
```sh
# Help Guile v2.2.x find packages in /usr/local
export GUILE_LOAD_PATH="/usr/local/share/guile/site/2.2${GUILE_LOAD_PATH:+:}$GUILE_LOAD_PATH"
export GUILE_LOAD_COMPILED_PATH="/usr/local/lib/guile/2.2/site-ccache${GUILE_LOAD_COMPILED_PATH:+:}$GUILE_COMPILED_LOAD_PATH"
```
Note that these environment variables are used to check for packages during
`./configure`, so they should be set as soon as possible should you want to use
a prefix other than `/usr`.
<!-- ##### Example: Consistently using Guile 3.0 on Ubuntu -->
<!-- For example, on Ubuntu, if you choose to use Guile 3.0 and install the -->
<!-- `guile-3.0` package, you want to make sure that if you also want to install the -->
<!-- `guile-git` package with `apt` that said `guile-git` package was built for Guile -->
<!-- v3.0. This can be checked by invoking the following: -->
<!-- ``` -->
<!-- apt update -->
<!-- apt show guile-git -->
<!-- ``` -->
#### Building and installing source-built packages
***IMPORTANT**: A few dependencies have non-obvious quirks/erratas which are documented in the
sub-sections immediately below. Please read these sections before proceeding to
build and install these packages.*
Although you should always refer to the README or INSTALL files for the most
accurate information, most of these dependencies use autoconf-style build
systems (check if there's a `configure.ac` file), and will likely do the right
thing with the following:
Clone the repository and check out the latest release:
```sh
git clone <git-repo-of-dependency>/<dependency>.git
cd <dependency>
git tag -l # check for the latest release
git checkout <latest-release>
```
For autoconf-based build systems (if `./autogen.sh` or `configure.ac` exists at
the root of the repository):
```sh
./autogen.sh || autoreconf -vfi
./configure --prefix=<prefix>
make
sudo make install
```
For CMake-based build systems (if `CMakeLists.txt` exists at the root of the
repository):
```sh
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=<prefix>
sudo cmake --build . --target install
```
If you choose not to specify exactly `--prefix=/usr` to `./configure`, please
make sure you've carefully read the [previous section] on mixing distribution
packages and source-built packages.
##### Binding packages require `-dev`-suffixed packages
Relevant for:
- Everyone
When building bindings, the `-dev`-suffixed version of the original package
needs to be installed. For example, building `Guile-zlib` on Debian-derived
distributions requires that `zlib1g-dev` is installed.
When using bindings, the `-dev`-suffixed version of the original package still
needs to be installed. This is particularly problematic when distribution
packages are mispackaged like `guile-sqlite3` is in Ubuntu Focal such that
installing `guile-sqlite3` does not automatically install `libsqlite3-dev` as a
dependency.
Below is a list of relevant Guile bindings and their corresponding `-dev`
packages in Debian at the time of writing.
| Guile binding package | -dev Debian package |
|-----------------------|---------------------|
| guile-gcrypt | libgcrypt-dev |
| guile-git | libgit2-dev |
| guile-lzlib | liblz-dev |
| guile-ssh | libssh-dev |
| guile-sqlite3 | libsqlite3-dev |
| guile-zlib | zlib1g-dev |
##### `guile-git` actually depends on `libgit2 >= 1.1`
Relevant for:
- Those building `guile-git` from source against `libgit2 < 1.1`
- Those installing `guile-git` from their distribution where `guile-git` is
built against `libgit2 < 1.1`
As of v0.4.0, `guile-git` claims to only require `libgit2 >= 0.28.0`, however,
it actually requires `libgit2 >= 1.1`, otherwise, it will be confused by a
reference of `origin/keyring`: instead of interpreting the reference as "the
'keyring' branch of the 'origin' remote", the reference is interpreted as "the
branch literally named 'origin/keyring'"
This is especially notable because Ubuntu Focal packages `libgit2 v0.28.4`, and
`guile-git` is built against it.
Should you be in this situation, you need to build both `libgit2 v1.1.x` and
`guile-git` from source.
Source: http://logs.guix.gnu.org/guix/2020-11-12.log#232527
##### `{scheme,guile}-bytestructures` v1.0.8 and v1.0.9 are broken for Guile v2.2
Relevant for:
- Those building `{scheme,guile}-bytestructures` from source against Guile v2.2
Commit
[707eea3](https://github.com/TaylanUB/scheme-bytestructures/commit/707eea3a85e1e375e86702229ebf73d496377669)
introduced a regression for Guile v2.2 and was first included in v1.0.8, this
was later corrected in commit
[ec9a721](https://github.com/TaylanUB/scheme-bytestructures/commit/ec9a721957c17bcda13148f8faa5f06934431ff7)
and included in v1.1.0.
TL;DR If you decided to use Guile v2.2, do not use `{scheme,guile}-bytestructures` v1.0.8 or v1.0.9.
### Building and Installing Guix itself
Start by cloning Guix:
```
git clone https://git.savannah.gnu.org/git/guix.git
cd guix
```
You will likely want to build the latest release, however, if the latest release
when you're reading this is still 1.2.0 then you may want to use 95aca29 instead
to avoid a problem in the GnuTLS test suite.
```
git branch -a -l 'origin/version-*' # check for the latest release
git checkout <latest-release>
```
Bootstrap the build system:
```
./bootstrap
```
Configure with the recommended `--localstatedir` flag:
```
./configure --localstatedir=/var
```
Note: If you intend to hack on Guix in the future, you will need to supply the
same `--localstatedir=` flag for all future Guix `./configure` invocations. See
the last paragraph of this
[section](https://guix.gnu.org/manual/en/html_node/Requirements.html) for more
details.
Build Guix (this will take a while):
```
make -j$(nproc)
```
Install Guix:
```
sudo make install
```
### Post-"build from source" Setup
#### Creating and starting a `guix-daemon-original` service with a fixed `argv[0]`
At this point, guix will be installed to `${bindir}`, which is likely
`/usr/local/bin` if you did not override directory variables at
`./configure`-time. More information on standard Automake directory variables
can be found
[here](https://www.gnu.org/software/automake/manual/html_node/Standard-Directory-Variables.html).
However, the Guix init scripts and service configurations for Upstart, systemd,
SysV, and OpenRC are installed (in `${libdir}`) to launch
`${localstatedir}/guix/profiles/per-user/root/current-guix/bin/guix-daemon`,
which does not yet exist, and will only exist after [`root` performs their first
`guix pull`](#guix-pull-as-root).
We need to create a `-original` version of these init scripts that's pointed to
the binaries we just built and `make install`'ed in `${bindir}` (normally,
`/usr/local/bin`).
Example for `systemd`, run as `root`:
```sh
# Create guix-daemon-original.service by modifying guix-daemon.service
libdir=# set according to your PREFIX (default is /usr/local/lib)
bindir="$(dirname $(command -v guix-daemon))"
sed -E -e "s|/\S*/guix/profiles/per-user/root/current-guix/bin/guix-daemon|${bindir}/guix-daemon|" "${libdir}"/systemd/system/guix-daemon.service > /etc/systemd/system/guix-daemon-original.service
chmod 664 /etc/systemd/system/guix-daemon-original.service
# Make systemd recognize the new service
systemctl daemon-reload
# Make sure that the non-working guix-daemon.service is stopped and disabled
systemctl stop guix-daemon
systemctl disable guix-daemon
# Make sure that the working guix-daemon-original.service is started and enabled
systemctl enable guix-daemon-original
systemctl start guix-daemon-original
```
#### Creating `guix-daemon` users / groups
Please see the [relevant
section](https://guix.gnu.org/manual/en/html_node/Build-Environment-Setup.html)
in the Guix Reference Manual for more details.
## Optional setup
At this point, you are set up to [use Guix to build DigiByte
Core](./README.md#usage). However, if you want to polish your setup a bit and
make it "what Guix intended", then read the next few subsections.
### Add an `/etc/profile.d` entry
This section definitely does not apply to you if you installed Guix using:
1. The shell installer script
2. fanquake's Docker image
3. Debian's `guix` package
#### Background
Although Guix knows how to update itself and its packages, it does so in a
non-invasive way (it does not modify `/usr/local/bin/guix`).
Instead, it does the following:
- After a `guix pull`, it updates
`/var/guix/profiles/per-user/$USER/current-guix`, and creates a symlink
targeting this directory at `$HOME/.config/guix/current`
- After a `guix install`, it updates
`/var/guix/profiles/per-user/$USER/guix-profile`, and creates a symlink
targeting this directory at `$HOME/.guix-profile`
Therefore, in order for these operations to affect your shell/desktop sessions
(and for the principle of least astonishment to hold), their corresponding
directories have to be added to well-known environment variables like `$PATH`,
`$INFOPATH`, `$XDG_DATA_DIRS`, etc.
In other words, if `$HOME/.config/guix/current/bin` does not exist in your
`$PATH`, a `guix pull` will have no effect on what `guix` you are using. Same
goes for `$HOME/.guix-profile/bin`, `guix install`, and installed packages.
Helpfully, after a `guix pull` or `guix install`, a message will be printed like
so:
```
hint: Consider setting the necessary environment variables by running:
GUIX_PROFILE="$HOME/.guix-profile"
. "$GUIX_PROFILE/etc/profile"
Alternately, see `guix package --search-paths -p "$HOME/.guix-profile"'.
```
However, this is somewhat tedious to do for both `guix pull` and `guix install`
for each user on the system that wants to properly use `guix`. I recommend that
you instead add an entry to `/etc/profile.d` instead. This is done by default
when installing the Debian package later than 1.2.0-4 and when using the shell
script installer.
#### Instructions
Create `/etc/profile.d/guix.sh` with the following content:
```sh
# _GUIX_PROFILE: `guix pull` profile
_GUIX_PROFILE="$HOME/.config/guix/current"
if [ -L $_GUIX_PROFILE ]; then
export PATH="$_GUIX_PROFILE/bin${PATH:+:}$PATH"
# Export INFOPATH so that the updated info pages can be found
# and read by both /usr/bin/info and/or $GUIX_PROFILE/bin/info
# When INFOPATH is unset, add a trailing colon so that Emacs
# searches 'Info-default-directory-list'.
export INFOPATH="$_GUIX_PROFILE/share/info:$INFOPATH"
fi
# GUIX_PROFILE: User's default profile
GUIX_PROFILE="$HOME/.guix-profile"
[ -L $GUIX_PROFILE ] || return
GUIX_LOCPATH="$GUIX_PROFILE/lib/locale"
export GUIX_PROFILE GUIX_LOCPATH
[ -f "$GUIX_PROFILE/etc/profile" ] && . "$GUIX_PROFILE/etc/profile"
# set XDG_DATA_DIRS to include Guix installations
export XDG_DATA_DIRS="$GUIX_PROFILE/share:${XDG_DATA_DIRS:-/usr/local/share/:/usr/share/}"
```
Please note that this will not take effect until the next shell or desktop
session (log out and log back in).
### `guix pull` as root
Before you do this, you need to read the section on [choosing your security
model][security-model] and adjust `guix` and `guix-daemon` flags according to
your choice, as invoking `guix pull` may pull substitutes from substitute
servers (which you may not want).
As mentioned in a previous section, Guix expects
`${localstatedir}/guix/profiles/per-user/root/current-guix` to be populated with
`root`'s Guix profile, `guix pull`-ed and built by some former version of Guix.
However, this is not the case when we build from source. Therefore, we need to
perform a `guix pull` as `root`:
```sh
sudo --login guix pull --branch=version-<latest-release-version>
# or
sudo --login guix pull --commit=<particular-commit>
```
`guix pull` is quite a long process (especially if you're using
`--no-substitute`). If you encounter build problems, please refer to the
[troubleshooting section](#troubleshooting).
Note that running a bare `guix pull` with no commit or branch specified will
pull the latest commit on Guix's master branch, which is likely fine, but not
recommended.
If you installed Guix from source, you may get an error like the following:
```sh
error: while creating symlink '/root/.config/guix/current' No such file or directory
```
To resolve this, simply:
```
sudo mkdir -p /root/.config/guix
```
Then try the `guix pull` command again.
After the `guix pull` finishes successfully,
`${localstatedir}/guix/profiles/per-user/root/current-guix` should be populated.
#### Using the newly-pulled `guix` by restarting the daemon
Depending on how you installed Guix, you should now make sure that your init
scripts and service configurations point to the newly-pulled `guix-daemon`.
##### If you built Guix from source
If you followed the instructions for [fixing argv\[0\]][fix-argv0], you can now
do the following:
```sh
systemctl stop guix-daemon-original
systemctl disable guix-daemon-original
systemctl enable guix-daemon
systemctl start guix-daemon
```
##### If you installed Guix via the Debian/Ubuntu distribution packages
You will need to create a `guix-daemon-latest` service which points to the new
`guix` rather than a pinned one.
```sh
# Create guix-daemon-latest.service by modifying guix-daemon.service
sed -E -e "s|/usr/bin/guix-daemon|/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon|" /etc/systemd/system/guix-daemon.service > /lib/systemd/system/guix-daemon-latest.service
chmod 664 /lib/systemd/system/guix-daemon-latest.service
# Make systemd recognize the new service
systemctl daemon-reload
# Make sure that the old guix-daemon.service is stopped and disabled
systemctl stop guix-daemon
systemctl disable guix-daemon
# Make sure that the new guix-daemon-latest.service is started and enabled
systemctl enable guix-daemon-latest
systemctl start guix-daemon-latest
```
##### If you installed Guix via lantw44's Arch Linux AUR package
At the time of writing (July 5th, 2021) the systemd unit for "updated Guix" is
`guix-daemon-latest.service`, therefore, you should do the following:
```sh
systemctl stop guix-daemon
systemctl disable guix-daemon
systemctl enable guix-daemon-latest
systemctl start guix-daemon-latest
```
##### Otherwise...
Simply do:
```sh
systemctl restart guix-daemon
```
### Checking everything
If you followed all the steps above to make your Guix setup "prim and proper,"
you can check that you did everything properly by running through this
checklist.
1. `/etc/profile.d/guix.sh` should exist and be sourced at each shell login
2. `guix describe` should not print `guix describe: error: failed to determine
origin`, but rather something like:
```
Generation 38 Feb 22 2021 16:39:31 (current)
guix f350df4
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: version-1.2.0
commit: f350df405fbcd5b9e27e6b6aa500da7f101f41e7
```
3. `guix-daemon` should be running from `${localstatedir}/guix/profiles/per-user/root/current-guix`
# Troubleshooting
## Derivation failed to build
When you see a build failure like below:
```
building /gnu/store/...-foo-3.6.12.drv...
/ 'check' phasenote: keeping build directory `/tmp/guix-build-foo-3.6.12.drv-0'
builder for `/gnu/store/...-foo-3.6.12.drv' failed with exit code 1
build of /gnu/store/...-foo-3.6.12.drv failed
View build log at '/var/log/guix/drvs/../...-foo-3.6.12.drv.bz2'.
cannot build derivation `/gnu/store/...-qux-7.69.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-bar-3.16.5.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-baz-2.0.5.drv': 1 dependencies couldn't be built
guix time-machine: error: build of `/gnu/store/...-baz-2.0.5.drv' failed
```
It means that `guix` failed to build a package named `foo`, which was a
dependency of `qux`, `bar`, and `baz`. Importantly, note that the last "failed"
line is not necessarily the root cause, the first "failed" line is.
Most of the time, the build failure is due to a spurious test failure or the
package's build system/test suite breaking when running multi-threaded. To
rebuild _just_ this derivation in a single-threaded fashion (please don't forget
to add other `guix` flags like `--no-substitutes` as appropriate):
```sh
$ guix build --cores=1 /gnu/store/...-foo-3.6.12.drv
```
If the single-threaded rebuild did not succeed, you may need to dig deeper.
You may view `foo`'s build logs in `less` like so (please replace paths with the
path you see in the build failure output):
```sh
$ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
```
`foo`'s build directory is also preserved and available at
`/tmp/guix-build-foo-3.6.12.drv-0`. However, if you fail to build `foo` multiple
times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
failure output for the most accurate, up-to-date information.
### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
which rejects characters not present in the UTF-8 character code set. An example
is ZFS with the utf8only=on option set.
More information: https://bugs.python.org/issue37584
### GnuTLS: test-suite FAIL: status-request-revoked
*The derivation is likely identified by: `/gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv`*
This unfortunate error is most common for non-substitute builders who installed
Guix v1.2.0. The problem stems from the fact that one of GnuTLS's tests uses a
hardcoded certificate which expired on 2020-10-24.
What's more unfortunate is that this GnuTLS derivation is somewhat special in
Guix's dependency graph and is not affected by the package transformation flags
like `--without-tests=`.
The easiest solution for those encountering this problem is to install a newer
version of Guix. However, there are ways to work around this issue:
#### Workaround 1: Using substitutes for this single derivation
If you've authorized the official Guix build farm's key (more info
[here](./README.md#step-1-authorize-the-signing-keys)), then you can use
substitutes just for this single derivation by invoking the following:
```sh
guix build --substitute-urls="https://ci.guix.gnu.org" /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
```
See [this section](./README.md#removing-authorized-keys) for instructions on how
to remove authorized keys if you don't want to keep the build farm's key
authorized.
#### Workaround 2: Temporarily setting the system clock back
This workaround was described [here](https://issues.guix.gnu.org/44559#5).
Basically:
1. Turn off networking
2. Turn off NTP
3. Set system time to 2020-10-01
4. guix build --no-substitutes /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
5. Set system time back to accurate current time
6. Turn NTP back on
7. Turn networking back on
### coreutils: FAIL: tests/tail-2/inotify-dir-recreate
The inotify-dir-create test fails on "remote" filesystems such as overlayfs
(Docker's default filesystem) due to the filesystem being mistakenly recognized
as non-remote.
A relatively easy workaround to this is to make sure that a somewhat traditional
filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds). For
Docker users, this might mean [using a volume][docker/volumes], [binding
mounting][docker/bind-mnt] from host, or (for those with enough RAM and swap)
[mounting a tmpfs][docker/tmpfs] using the `--tmpfs` flag.
Please see the following links for more details:
- An upstream coreutils bug has been filed: [debbugs#47940](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47940)
- A Guix bug detailing the underlying problem has been filed: [guix-issues#47935](https://issues.guix.gnu.org/47935)
- A commit to skip this test in Guix has been merged into the core-updates branch:
[savannah/guix@6ba1058](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6ba1058df0c4ce5611c2367531ae5c3cdc729ab4)
[install-script]: #options-1-and-2-using-the-official-shell-installer-script-or-binary-tarball
[install-bin-tarball]: #options-1-and-2-using-the-official-shell-installer-script-or-binary-tarball
[install-fanquake-docker]: #option-3-using-fanquakes-docker-image
[install-distro-pkg]: #option-4-using-a-distribution-maintained-package
[install-source]: #option-5-building-from-source
[fix-argv0]: #creating-and-starting-a-guix-daemon-original-service-with-a-fixed-argv0
[security-model]: ./README.md#choosing-your-security-model
[docker/volumes]: https://docs.docker.com/storage/volumes/
[docker/bind-mnt]: https://docs.docker.com/storage/bind-mounts/
[docker/tmpfs]: https://docs.docker.com/storage/tmpfs/

485
contrib/guix/README.md Normal file
View file

@ -0,0 +1,485 @@
# Bootstrappable DigiByte Core Builds
This directory contains the files necessary to perform bootstrappable DigiByte
Core builds.
[Bootstrappability][b17e] furthers our binary security guarantees by allowing us
to _audit and reproduce_ our toolchain instead of blindly _trusting_ binary
downloads.
We achieve bootstrappability by using Guix as a functional package manager.
# Requirements
Conservatively, a x86_64 machine with:
- 16GB of free disk space on the partition that /gnu/store will reside in
- 8GB of free disk space **per platform triple** you're planning on building
(see the `HOSTS` [environment variable description][env-vars-list])
# Installation and Setup
If you don't have Guix installed and set up, please follow the instructions in
[INSTALL.md](./INSTALL.md)
# Usage
If you haven't considered your security model yet, please read [the relevant
section](#choosing-your-security-model) before proceeding to perform a build.
## Making the Xcode SDK available for macOS cross-compilation
In order to perform a build for macOS (which is included in the default set of
platform triples to build), you'll need to extract the macOS SDK tarball using
tools found in the [`macdeploy` directory](../macdeploy/README.md).
You can then either point to the SDK using the `SDK_PATH` environment variable:
```sh
# Extract the SDK tarball to /path/to/parent/dir/of/extracted/SDK/Xcode-<foo>-<bar>-extracted-SDK-with-libcxx-headers
tar -C /path/to/parent/dir/of/extracted/SDK -xaf /path/to/Xcode-<foo>-<bar>-extracted-SDK-with-libcxx-headers.tar.gz
# Indicate where to locate the SDK tarball
export SDK_PATH=/path/to/parent/dir/of/extracted/SDK
```
or extract it into `depends/SDKs`:
```sh
mkdir -p depends/SDKs
tar -C depends/SDKs -xaf /path/to/SDK/tarball
```
## Building
*The author highly recommends at least reading over the [common usage patterns
and examples](#common-guix-build-invocation-patterns-and-examples) section below
before starting a build. For a full list of customization options, see the
[recognized environment variables][env-vars-list] section.*
To build DigiByte Core reproducibly with all default options, invoke the
following from the top of a clean repository:
```sh
./contrib/guix/guix-build
```
## Codesigning build outputs
The `guix-codesign` command attaches codesignatures (produced by codesigners) to
existing non-codesigned outputs. Please see the [release process
documentation](/doc/release-process.md) for more context.
It respects many of the same environment variable flags as `guix-build`, with 2
crucial differences:
1. Since only Windows and macOS build outputs require codesigning, the `HOSTS`
environment variable will have a sane default value of `x86_64-w64-mingw32
x86_64-apple-darwin18` instead of all the platforms.
2. The `guix-codesign` command ***requires*** a `DETACHED_SIGS_REPO` flag.
* _**DETACHED_SIGS_REPO**_
Set the directory where detached codesignatures can be found for the current
DigiByte Core version being built.
_REQUIRED environment variable_
An invocation with all default options would look like:
```
env DETACHED_SIGS_REPO=<path/to/digibyte-detached-sigs> ./contrib/guix-codesign
```
## Cleaning intermediate work directories
By default, `guix-build` leaves all intermediate files or "work directories"
(e.g. `depends/work`, `guix-build-*/distsrc-*`) intact at the end of a build so
that they are available to the user (to aid in debugging, etc.). However, these
directories usually take up a large amount of disk space. Therefore, a
`guix-clean` convenience script is provided which cleans the current `git`
worktree to save disk space:
```
./contrib/guix/guix-clean
```
## Attesting to build outputs
Much like how Gitian build outputs are attested to in a `gitian.sigs`
repository, Guix build outputs are attested to in the [`guix.sigs`
repository](https://github.com/digibyte-core/guix.sigs).
After you've cloned the `guix.sigs` repository, to attest to the current
worktree's commit/tag:
```
env GUIX_SIGS_REPO=<path/to/guix.sigs> SIGNER=<gpg-key-name> ./contrib/guix/guix-attest
```
See `./contrib/guix/guix-attest --help` for more information on the various ways
`guix-attest` can be invoked.
## Verifying build output attestations
After at least one other signer has uploaded their signatures to the `guix.sigs`
repository:
```
git -C <path/to/guix.sigs> pull
env GUIX_SIGS_REPO=<path/to/guix.sigs> ./contrib/guix/guix-verify
```
## Common `guix-build` invocation patterns and examples
### Keeping caches and SDKs outside of the worktree
If you perform a lot of builds and have a bunch of worktrees, you may find it
more efficient to keep the depends tree's download cache, build cache, and SDKs
outside of the worktrees to avoid duplicate downloads and unnecessary builds. To
help with this situation, the `guix-build` script honours the `SOURCES_PATH`,
`BASE_CACHE`, and `SDK_PATH` environment variables and will pass them on to the
depends tree so that you can do something like:
```sh
env SOURCES_PATH="$HOME/depends-SOURCES_PATH" BASE_CACHE="$HOME/depends-BASE_CACHE" SDK_PATH="$HOME/macOS-SDKs" ./contrib/guix/guix-build
```
Note that the paths that these environment variables point to **must be
directories**, and **NOT symlinks to directories**.
See the [recognized environment variables][env-vars-list] section for more
details.
### Building a subset of platform triples
Sometimes you only want to build a subset of the supported platform triples, in
which case you can override the default list by setting the space-separated
`HOSTS` environment variable:
```sh
env HOSTS='x86_64-w64-mingw32 x86_64-apple-darwin18' ./contrib/guix/guix-build
```
See the [recognized environment variables][env-vars-list] section for more
details.
### Controlling the number of threads used by `guix` build commands
Depending on your system's RAM capacity, you may want to decrease the number of
threads used to decrease RAM usage or vice versa.
By default, the scripts under `./contrib/guix` will invoke all `guix` build
commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
specified. However, astute manual readers will also notice that `guix` build
commands also accept a `--max-jobs=` flag (which defaults to 1 if unspecified).
Here is the difference between `--cores=` and `--max-jobs=`:
> Note: When I say "derivation," think "package"
`--cores=`
- controls the number of CPU cores to build each derivation. This is the value
passed to `make`'s `--jobs=` flag.
`--max-jobs=`
- controls how many derivations can be built in parallel
- defaults to 1
Therefore, the default is for `guix` build commands to build one derivation at a
time, utilizing `$JOBS` threads.
Specifying the `$JOBS` environment variable will only modify `--cores=`, but you
can also modify the value for `--max-jobs=` by specifying
`$ADDITIONAL_GUIX_COMMON_FLAGS`. For example, if you have a LOT of memory, you
may want to set:
```sh
export ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
```
Which allows for a maximum of 8 derivations to be built at the same time, each
utilizing `$JOBS` threads.
Or, if you'd like to avoid spurious build failures caused by issues with
parallelism within a single package, but would still like to build multiple
packages when the dependency graph allows for it, you may want to try:
```sh
export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
```
See the [recognized environment variables][env-vars-list] section for more
details.
## Recognized environment variables
* _**HOSTS**_
Override the space-separated list of platform triples for which to perform a
bootstrappable build.
_(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu
riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
x86\_64-w64-mingw32 x86\_64-apple-darwin arm64-apple-darwin")_
* _**SOURCES_PATH**_
Set the depends tree download cache for sources. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary redownloading of package sources.
The path that this environment variable points to **must be a directory**, and
**NOT a symlink to a directory**.
* _**BASE_CACHE**_
Set the depends tree cache for built packages. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary building of packages.
The path that this environment variable points to **must be a directory**, and
**NOT a symlink to a directory**.
* _**SDK_PATH**_
Set the path where _extracted_ SDKs can be found. This is passed through to
the depends tree. Note that this is should be set to the _parent_ directory of
the actual SDK (e.g. `SDK_PATH=$HOME/Downloads/macOS-SDKs` instead of
`$HOME/Downloads/macOS-SDKs/Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers`).
The path that this environment variable points to **must be a directory**, and
**NOT a symlink to a directory**.
* _**JOBS**_
Override the number of jobs to run simultaneously, you might want to do so on
a memory-limited machine. This may be passed to:
- `guix` build commands as in `guix environment --cores="$JOBS"`
- `make` as in `make --jobs="$JOBS"`
- `xargs` as in `xargs -P"$JOBS"`
See [here](#controlling-the-number-of-threads-used-by-guix-build-commands) for
more details.
_(defaults to the value of `nproc` outside the container)_
* _**SOURCE_DATE_EPOCH**_
Override the reference UNIX timestamp used for bit-for-bit reproducibility,
the variable name conforms to [standard][r12e/source-date-epoch].
_(defaults to the output of `$(git log --format=%at -1)`)_
* _**V**_
If non-empty, will pass `V=1` to all `make` invocations, making `make` output
verbose.
Note that any given value is ignored. The variable is only checked for
emptiness. More concretely, this means that `V=` (setting `V` to the empty
string) is interpreted the same way as not setting `V` at all, and that `V=0`
has the same effect as `V=1`.
* _**SUBSTITUTE_URLS**_
A whitespace-delimited list of URLs from which to download pre-built packages.
A URL is only used if its signing key is authorized (refer to the [substitute
servers section](#option-1-building-with-substitutes) for more details).
* _**ADDITIONAL_GUIX_COMMON_FLAGS**_
Additional flags to be passed to all `guix` commands.
* _**ADDITIONAL_GUIX_TIMEMACHINE_FLAGS**_
Additional flags to be passed to `guix time-machine`.
* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
Additional flags to be passed to the invocation of `guix environment` inside
`guix time-machine`.
# Choosing your security model
No matter how you installed Guix, you need to decide on your security model for
building packages with Guix.
Guix allows us to achieve better binary security by using our CPU time to build
everything from scratch. However, it doesn't sacrifice user choice in pursuit of
this: users can decide whether or not to use **substitutes** (pre-built
packages).
## Option 1: Building with substitutes
### Step 1: Authorize the signing keys
Depending on the installation procedure you followed, you may have already
authorized the Guix build farm key. In particular, the official shell installer
script asks you if you want the key installed, and the debian distribution
package authorized the key during installation.
You can check the current list of authorized keys at `/etc/guix/acl`.
At the time of writing, a `/etc/guix/acl` with just the Guix build farm key
authorized looks something like:
```lisp
(acl
(entry
(public-key
(ecc
(curve Ed25519)
(q #8D156F295D24B0D9A86FA5741A840FF2D24F60F7B6C4134814AD55625971B394#)
)
)
(tag
(guix import)
)
)
)
```
If you've determined that the official Guix build farm key hasn't been
authorized, and you would like to authorize it, run the following as root:
```
guix archive --authorize < /var/guix/profiles/per-user/root/current-guix/share/guix/ci.guix.gnu.org.pub
```
If
`/var/guix/profiles/per-user/root/current-guix/share/guix/ci.guix.gnu.org.pub`
doesn't exist, try:
```sh
guix archive --authorize < <PREFIX>/share/guix/ci.guix.gnu.org.pub
```
Where `<PREFIX>` is likely:
- `/usr` if you installed from a distribution package
- `/usr/local` if you installed Guix from source and didn't supply any
prefix-modifying flags to Guix's `./configure`
For dongcarl's substitute server at https://guix.carldong.io, run as root:
```sh
wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
```
#### Removing authorized keys
To remove previously authorized keys, simply edit `/etc/guix/acl` and remove the
`(entry (public-key ...))` entry.
### Step 2: Specify the substitute servers
Once its key is authorized, the official Guix build farm at
https://ci.guix.gnu.org is automatically used unless the `--no-substitutes` flag
is supplied. This default list of substitute servers is overridable both on a
`guix-daemon` level and when you invoke `guix` commands. See examples below for
the various ways of adding dongcarl's substitute server after having [authorized
his signing key](#authorize-the-signing-keys).
Change the **default list** of substitute servers by starting `guix-daemon` with
the `--substitute-urls` option (you will likely need to edit your init script):
```sh
guix-daemon <cmd> --substitute-urls='https://guix.carldong.io https://ci.guix.gnu.org'
```
Override the default list of substitute servers by passing the
`--substitute-urls` option for invocations of `guix` commands:
```sh
guix <cmd> --substitute-urls='https://guix.carldong.io https://ci.guix.gnu.org'
```
For scripts under `./contrib/guix`, set the `SUBSTITUTE_URLS` environment
variable:
```sh
export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
```
## Option 2: Disabling substitutes on an ad-hoc basis
If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
like in the following snippet. The first build will take a while, but the
resulting packages will be cached for future builds.
For direct invocations of `guix`:
```sh
guix <cmd> --no-substitutes
```
For the scripts under `./contrib/guix/`:
```sh
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
```
## Option 3: Disabling substitutes by default
`guix-daemon` accepts a `--no-substitutes` flag, which will make sure that,
unless otherwise overridden by a command line invocation, no substitutes will be
used.
If you start `guix-daemon` using an init script, you can edit said script to
supply this flag.
# Purging/Uninstalling Guix
In the extraordinarily rare case where you messed up your Guix installation in
an irreversible way, you may want to completely purge Guix from your system and
start over.
1. Uninstall Guix itself according to the way you installed it. (e.g. `sudo apt
purge guix` for Ubuntu packaging, `sudo make uninstall` for
built-from-source).
2. Remove all build users and groups
You may check for relevant users and groups using:
```
getent passwd | grep guix
getent group | grep guix
```
Then, you may remove users and groups using:
```
sudo userdel <user>
sudo groupdel <group>
```
3. Remove all possible Guix-related directories
- `/var/guix/`
- `/var/log/guix/`
- `/gnu/`
- `/etc/guix/`
- `/home/*/.config/guix/`
- `/home/*/.cache/guix/`
- `/home/*/.guix-profile/`
- `/root/.config/guix/`
- `/root/.cache/guix/`
- `/root/.guix-profile/`
[b17e]: http://bootstrappable.org/
[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html
[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html
[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
[guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html
[debian/guix-bullseye]: https://packages.debian.org/bullseye/guix
[ubuntu/guix-hirsute]: https://packages.ubuntu.com/hirsute/guix
[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix
[env-vars-list]: #recognized-environment-variables

258
contrib/guix/guix-attest Executable file
View file

@ -0,0 +1,258 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
# Source the common prelude, which:
# 1. Checks if we're at the top directory of the DigiByte Core repository
# 2. Defines a few common functions and variables
#
# shellcheck source=libexec/prelude.bash
source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
###################
## Sanity Checks ##
###################
################
# Required non-builtin commands should be invokable
################
check_tools cat env basename mkdir diff sort
if [ -z "$NO_SIGN" ]; then
check_tools gpg
fi
################
# Required env vars should be non-empty
################
cmd_usage() {
cat <<EOF
Synopsis:
env GUIX_SIGS_REPO=<path/to/guix.sigs> \\
SIGNER=GPG_KEY_NAME[=SIGNER_NAME] \\
[ NO_SIGN=1 ]
./contrib/guix/guix-attest
Example w/o overriding signing name:
env GUIX_SIGS_REPO=/home/achow101/guix.sigs \\
SIGNER=achow101 \\
./contrib/guix/guix-attest
Example overriding signing name:
env GUIX_SIGS_REPO=/home/dongcarl/guix.sigs \\
SIGNER=0x96AB007F1A7ED999=dongcarl \\
./contrib/guix/guix-attest
Example w/o signing, just creating SHA256SUMS:
env GUIX_SIGS_REPO=/home/achow101/guix.sigs \\
SIGNER=achow101 \\
NO_SIGN=1 \\
./contrib/guix/guix-attest
EOF
}
if [ -z "$GUIX_SIGS_REPO" ] || [ -z "$SIGNER" ]; then
cmd_usage
exit 1
fi
################
# GUIX_SIGS_REPO should exist as a directory
################
if [ ! -d "$GUIX_SIGS_REPO" ]; then
cat << EOF
ERR: The specified GUIX_SIGS_REPO is not an existent directory:
'$GUIX_SIGS_REPO'
Hint: Please clone the guix.sigs repository and point to it with the
GUIX_SIGS_REPO environment variable.
EOF
cmd_usage
exit 1
fi
################
# The key specified in SIGNER should be usable
################
IFS='=' read -r gpg_key_name signer_name <<< "$SIGNER"
if [ -z "${signer_name}" ]; then
signer_name="$gpg_key_name"
fi
if [ -z "$NO_SIGN" ] && ! gpg --dry-run --list-secret-keys "${gpg_key_name}" >/dev/null 2>&1; then
echo "ERR: GPG can't seem to find any key named '${gpg_key_name}'"
exit 1
fi
################
# We should be able to find at least one output
################
echo "Looking for build output SHA256SUMS fragments in ${OUTDIR_BASE}"
shopt -s nullglob
sha256sum_fragments=( "$OUTDIR_BASE"/*/SHA256SUMS.part ) # This expands to an array of directories...
shopt -u nullglob
noncodesigned_fragments=()
codesigned_fragments=()
if (( ${#sha256sum_fragments[@]} )); then
echo "Found build output SHA256SUMS fragments:"
for outdir in "${sha256sum_fragments[@]}"; do
echo " '$outdir'"
case "$outdir" in
"$OUTDIR_BASE"/*-codesigned/SHA256SUMS.part)
codesigned_fragments+=("$outdir")
;;
*)
noncodesigned_fragments+=("$outdir")
;;
esac
done
echo
else
echo "ERR: Could not find any build output SHA256SUMS fragments in ${OUTDIR_BASE}"
exit 1
fi
##############
## Attest ##
##############
# Usage: out_name $outdir
#
# HOST: The output directory being attested
#
out_name() {
basename "$(dirname "$1")"
}
shasum_already_exists() {
cat <<EOF
--
ERR: An ${1} file already exists for '${VERSION}' and attests
differently. You likely previously attested to a partial build (e.g. one
where you specified the HOST environment variable).
See the diff above for more context.
Hint: You may wish to remove the existing attestations and their signatures by
invoking:
rm '${PWD}/${1}'{,.asc}
Then try running this script again.
EOF
}
# Given a document with unix line endings (just <LF>) in stdin, make all lines
# end in <CR><LF> and make sure there's no trailing <LF> at the end of the file.
#
# This is necessary as cleartext signatures are calculated on text after their
# line endings are canonicalized.
#
# For more information:
# 1. https://security.stackexchange.com/a/104261
# 2. https://datatracker.ietf.org/doc/html/rfc4880#section-7.1
#
rfc4880_normalize_document() {
sed 's/$/\r/' | head -c -2
}
echo "Attesting to build outputs for version: '${VERSION}'"
echo ""
outsigdir="$GUIX_SIGS_REPO/$VERSION/$signer_name"
mkdir -p "$outsigdir"
(
cd "$outsigdir"
temp_noncodesigned="$(mktemp)"
trap 'rm -rf -- "$temp_noncodesigned"' EXIT
if (( ${#noncodesigned_fragments[@]} )); then
cat "${noncodesigned_fragments[@]}" \
| sort -u \
| sort -k2 \
| rfc4880_normalize_document \
> "$temp_noncodesigned"
if [ -e noncodesigned.SHA256SUMS ]; then
# The SHA256SUMS already exists, make sure it's exactly what we
# expect, error out if not
if diff -u noncodesigned.SHA256SUMS "$temp_noncodesigned"; then
echo "A noncodesigned.SHA256SUMS file already exists for '${VERSION}' and is up-to-date."
else
shasum_already_exists noncodesigned.SHA256SUMS
exit 1
fi
else
mv "$temp_noncodesigned" noncodesigned.SHA256SUMS
fi
else
echo "ERR: No noncodesigned outputs found for '${VERSION}', exiting..."
exit 1
fi
temp_codesigned="$(mktemp)"
trap 'rm -rf -- "$temp_codesigned"' EXIT
if (( ${#codesigned_fragments[@]} )); then
# Note: all.SHA256SUMS attests to all of $sha256sum_fragments, but is
# not needed if there are no $codesigned_fragments
cat "${sha256sum_fragments[@]}" \
| sort -u \
| sort -k2 \
| sed 's/$/\r/' \
| rfc4880_normalize_document \
> "$temp_codesigned"
if [ -e codesigned.SHA256SUMS ]; then
# The SHA256SUMS already exists, make sure it's exactly what we
# expect, error out if not
if diff -u all.SHA256SUMS "$temp_codesigned"; then
echo "An all.SHA256SUMS file already exists for '${VERSION}' and is up-to-date."
else
shasum_already_exists all.SHA256SUMS
exit 1
fi
else
mv "$temp_codesigned" codesigned.SHA256SUMS
fi
else
# It is fine to have the codesigned outputs be missing (perhaps the
# detached codesigs have not been published yet), just print a log
# message instead of erroring out
echo "INFO: No codesigned outputs found for '${VERSION}', skipping..."
fi
if [ -z "$NO_SIGN" ]; then
echo "Signing SHA256SUMS to produce SHA256SUMS.asc"
for i in *.SHA256SUMS; do
if [ ! -e "$i".asc ]; then
gpg --detach-sign \
--digest-algo sha256 \
--local-user "$gpg_key_name" \
--armor \
--output "$i".asc "$i"
else
echo "Signature already there"
fi
done
else
echo "Not signing SHA256SUMS as \$NO_SIGN is not empty"
fi
echo ""
)

477
contrib/guix/guix-build Executable file
View file

@ -0,0 +1,477 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
# Source the common prelude, which:
# 1. Checks if we're at the top directory of the DigiByte Core repository
# 2. Defines a few common functions and variables
#
# shellcheck source=libexec/prelude.bash
source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
###################
## SANITY CHECKS ##
###################
################
# Required non-builtin commands should be invocable
################
check_tools cat mkdir make getent curl git guix
################
# GUIX_BUILD_OPTIONS should be empty
################
#
# GUIX_BUILD_OPTIONS is an environment variable recognized by guix commands that
# can perform builds. This seems like what we want instead of
# ADDITIONAL_GUIX_COMMON_FLAGS, but the value of GUIX_BUILD_OPTIONS is actually
# _appended_ to normal command-line options. Meaning that they will take
# precedence over the command-specific ADDITIONAL_GUIX_<CMD>_FLAGS.
#
# This seems like a poor user experience. Thus we check for GUIX_BUILD_OPTIONS's
# existence here and direct users of this script to use our (more flexible)
# custom environment variables.
if [ -n "$GUIX_BUILD_OPTIONS" ]; then
cat << EOF
Error: Environment variable GUIX_BUILD_OPTIONS is not empty:
'$GUIX_BUILD_OPTIONS'
Unfortunately this script is incompatible with GUIX_BUILD_OPTIONS, please unset
GUIX_BUILD_OPTIONS and use ADDITIONAL_GUIX_COMMON_FLAGS to set build options
across guix commands or ADDITIONAL_GUIX_<CMD>_FLAGS to set build options for a
specific guix command.
See contrib/guix/README.md for more details.
EOF
exit 1
fi
################
# The git worktree should not be dirty
################
if ! git diff-index --quiet HEAD -- && [ -z "$FORCE_DIRTY_WORKTREE" ]; then
cat << EOF
ERR: The current git worktree is dirty, which may lead to broken builds.
Aborting...
Hint: To make your git worktree clean, You may want to:
1. Commit your changes,
2. Stash your changes, or
3. Set the 'FORCE_DIRTY_WORKTREE' environment variable if you insist on
using a dirty worktree
EOF
exit 1
fi
mkdir -p "$VERSION_BASE"
################
# Build directories should not exist
################
# Default to building for all supported HOSTs (overridable by environment)
export HOSTS="${HOSTS:-x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
x86_64-w64-mingw32
x86_64-apple-darwin arm64-apple-darwin}"
# Usage: distsrc_for_host HOST
#
# HOST: The current platform triple we're building for
#
distsrc_for_host() {
echo "${DISTSRC_BASE}/distsrc-${VERSION}-${1}"
}
# Accumulate a list of build directories that already exist...
hosts_distsrc_exists=""
for host in $HOSTS; do
if [ -e "$(distsrc_for_host "$host")" ]; then
hosts_distsrc_exists+=" ${host}"
fi
done
if [ -n "$hosts_distsrc_exists" ]; then
# ...so that we can print them out nicely in an error message
cat << EOF
ERR: Build directories for this commit already exist for the following platform
triples you're attempting to build, probably because of previous builds.
Please remove, or otherwise deal with them prior to starting another build.
Aborting...
Hint: To blow everything away, you may want to use:
$ ./contrib/guix/guix-clean
Specifically, this will remove all files without an entry in the index,
excluding the SDK directory, the depends download cache, the depends built
packages cache, the garbage collector roots for Guix environments, and the
output directory.
EOF
for host in $hosts_distsrc_exists; do
echo " ${host} '$(distsrc_for_host "$host")'"
done
exit 1
else
mkdir -p "$DISTSRC_BASE"
fi
################
# When building for darwin, the macOS SDK should exists
################
for host in $HOSTS; do
case "$host" in
*darwin*)
OSX_SDK="$(make -C "${PWD}/depends" --no-print-directory HOST="$host" print-OSX_SDK | sed 's@^[^=]\+=@@g')"
if [ -e "$OSX_SDK" ]; then
echo "Found macOS SDK at '${OSX_SDK}', using..."
else
echo "macOS SDK does not exist at '${OSX_SDK}', please place the extracted, untarred SDK there to perform darwin builds, exiting..."
exit 1
fi
;;
esac
done
################
# VERSION_BASE should have enough space
################
avail_KiB="$(df -Pk "$VERSION_BASE" | sed 1d | tr -s ' ' | cut -d' ' -f4)"
total_required_KiB=0
for host in $HOSTS; do
case "$host" in
*darwin*) required_KiB=440000 ;;
*mingw*) required_KiB=7600000 ;;
*) required_KiB=6400000 ;;
esac
total_required_KiB=$((total_required_KiB+required_KiB))
done
if (( total_required_KiB > avail_KiB )); then
total_required_GiB=$((total_required_KiB / 1048576))
avail_GiB=$((avail_KiB / 1048576))
echo "Performing a DigiByte Core Guix build for the selected HOSTS requires ${total_required_GiB} GiB, however, only ${avail_GiB} GiB is available. Please free up some disk space before performing the build."
exit 1
fi
################
# Check that we can connect to the guix-daemon
################
cat << EOF
Checking that we can connect to the guix-daemon...
Hint: If this hangs, you may want to try turning your guix-daemon off and on
again.
EOF
if ! guix gc --list-failures > /dev/null; then
cat << EOF
ERR: Failed to connect to the guix-daemon, please ensure that one is running and
reachable.
EOF
exit 1
fi
# Developer note: we could use `guix repl` for this check and run:
#
# (import (guix store)) (close-connection (open-connection))
#
# However, the internal API is likely to change more than the CLI invocation
################
# Services database must have basic entries
################
if ! getent services http https ftp; then
cat << EOF
ERR: Your system's C library can not find service database entries for at least
one of the following services: http, https, ftp.
Hint: Most likely, /etc/services does not exist yet (common for docker images
and minimal distros), or you don't have permissions to access it.
If /etc/services does not exist yet, you may want to install the
appropriate package for your distro which provides it.
On Debian/Ubuntu: netbase
On Arch Linux: iana-etc
For more information, see: getent(1), services(5)
EOF
fi
#########
# SETUP #
#########
# Determine the maximum number of jobs to run simultaneously (overridable by
# environment)
JOBS="${JOBS:-$(nproc)}"
# Usage: host_to_commonname HOST
#
# HOST: The current platform triple we're building for
#
host_to_commonname() {
case "$1" in
*darwin*) echo osx ;;
*mingw*) echo win ;;
*linux*) echo linux ;;
*) exit 1 ;;
esac
}
# Determine the reference time used for determinism (overridable by environment)
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
# Execute "$@" in a pinned, possibly older version of Guix, for reproducibility
# across time.
time-machine() {
# shellcheck disable=SC2086
guix time-machine --url=https://git.savannah.gnu.org/git/guix.git \
--commit=aa34d4d28dfe25ba47d5800d05000fb7221788c0 \
--cores="$JOBS" \
--keep-failed \
--fallback \
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \
-- "$@"
}
# Precious directories are those which should not be cleaned between successive
# guix builds
depends_precious_dir_names='SOURCES_PATH BASE_CACHE SDK_PATH'
precious_dir_names="${depends_precious_dir_names} OUTDIR_BASE PROFILES_BASE"
# Usage: contains IFS-SEPARATED-LIST ITEM
contains() {
for i in ${1}; do
if [ "$i" = "${2}" ]; then
return 0 # Found!
fi
done
return 1
}
# If the user explicitly specified a precious directory, create it so we
# can map it into the container
for precious_dir_name in $precious_dir_names; do
precious_dir_path="${!precious_dir_name}"
if [ -n "$precious_dir_path" ]; then
if [ ! -e "$precious_dir_path" ]; then
mkdir -p "$precious_dir_path"
elif [ -L "$precious_dir_path" ]; then
echo "ERR: ${precious_dir_name} cannot be a symbolic link"
exit 1
elif [ ! -d "$precious_dir_path" ]; then
echo "ERR: ${precious_dir_name} must be a directory"
exit 1
fi
fi
done
mkdir -p "$VAR_BASE"
# Record the _effective_ values of precious directories such that guix-clean can
# avoid clobbering them if appropriate.
#
# shellcheck disable=SC2046,SC2086
{
# Get depends precious dir definitions from depends
make -C "${PWD}/depends" \
--no-print-directory \
-- $(printf "print-%s\n" $depends_precious_dir_names)
# Get remaining precious dir definitions from the environment
for precious_dir_name in $precious_dir_names; do
precious_dir_path="${!precious_dir_name}"
if ! contains "$depends_precious_dir_names" "$precious_dir_name"; then
echo "${precious_dir_name}=${precious_dir_path}"
fi
done
} > "${VAR_BASE}/precious_dirs"
# Make sure an output directory exists for our builds
OUTDIR_BASE="${OUTDIR_BASE:-${VERSION_BASE}/output}"
mkdir -p "$OUTDIR_BASE"
# Download the depends sources now as we won't have internet access in the build
# container
for host in $HOSTS; do
make -C "${PWD}/depends" -j"$JOBS" download-"$(host_to_commonname "$host")" ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
done
# Usage: outdir_for_host HOST SUFFIX
#
# HOST: The current platform triple we're building for
#
outdir_for_host() {
echo "${OUTDIR_BASE}/${1}${2:+-${2}}"
}
# Usage: profiledir_for_host HOST SUFFIX
#
# HOST: The current platform triple we're building for
#
profiledir_for_host() {
echo "${PROFILES_BASE}/${1}${2:+-${2}}"
}
#########
# BUILD #
#########
# Function to be called when building for host ${1} and the user interrupts the
# build
int_trap() {
cat << EOF
** INT received while building ${1}, you may want to clean up the relevant
work directories (e.g. distsrc-*) before rebuilding
Hint: To blow everything away, you may want to use:
$ ./contrib/guix/guix-clean
Specifically, this will remove all files without an entry in the index,
excluding the SDK directory, the depends download cache, the depends built
packages cache, the garbage collector roots for Guix environments, and the
output directory.
EOF
}
# Deterministically build DigiByte Core
# shellcheck disable=SC2153
for host in $HOSTS; do
# Display proper warning when the user interrupts the build
trap 'int_trap ${host}' INT
(
# Required for 'contrib/guix/manifest.scm' to output the right manifest
# for the particular $HOST we're building for
export HOST="$host"
# shellcheck disable=SC2030
cat << EOF
INFO: Building ${VERSION:?not set} for platform triple ${HOST:?not set}:
...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set}
...running at most ${JOBS:?not set} jobs
...from worktree directory: '${PWD}'
...bind-mounted in container to: '/digibyte'
...in build directory: '$(distsrc_for_host "$HOST")'
...bind-mounted in container to: '$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")'
...outputting in: '$(outdir_for_host "$HOST")'
...bind-mounted in container to: '$(OUTDIR_BASE=/outdir-base && outdir_for_host "$HOST")'
EOF
# Run the build script 'contrib/guix/libexec/build.sh' in the build
# container specified by 'contrib/guix/manifest.scm'.
#
# Explanation of `guix environment` flags:
#
# --container run command within an isolated container
#
# Running in an isolated container minimizes build-time differences
# between machines and improves reproducibility
#
# --pure unset existing environment variables
#
# Same rationale as --container
#
# --no-cwd do not share current working directory with an
# isolated container
#
# When --container is specified, the default behavior is to share
# the current working directory with the isolated container at the
# same exact path (e.g. mapping '/home/satoshi/digibyte/' to
# '/home/satoshi/digibyte/'). This means that the $PWD inside the
# container becomes a source of irreproducibility. --no-cwd disables
# this behaviour.
#
# --share=SPEC for containers, share writable host file system
# according to SPEC
#
# --share="$PWD"=/digibyte
#
# maps our current working directory to /digibyte
# inside the isolated container, which we later cd
# into.
#
# While we don't want to map our current working directory to the
# same exact path (as this introduces irreproducibility), we do want
# it to be at a _fixed_ path _somewhere_ inside the isolated
# container so that we have something to build. '/digibyte' was
# chosen arbitrarily.
#
# ${SOURCES_PATH:+--share="$SOURCES_PATH"}
#
# make the downloaded depends sources path available
# inside the isolated container
#
# The isolated container has no network access as it's in a
# different network namespace from the main machine, so we have to
# make the downloaded depends sources available to it. The sources
# should have been downloaded prior to this invocation.
#
# --keep-failed keep build tree of failed builds
#
# When builds of the Guix environment itself (not DigiByte Core)
# fail, it is useful for the build tree to be kept for debugging
# purposes.
#
# ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"}
#
# fetch substitute from SUBSTITUTE_URLS if they are
# authorized
#
# Depending on the user's security model, it may be desirable to use
# substitutes (pre-built packages) from servers that the user trusts.
# Please read the README.md in the same directory as this file for
# more information.
#
# shellcheck disable=SC2086,SC2031
time-machine environment --manifest="${PWD}/contrib/guix/manifest.scm" \
--container \
--pure \
--no-cwd \
--share="$PWD"=/digibyte \
--share="$DISTSRC_BASE"=/distsrc-base \
--share="$OUTDIR_BASE"=/outdir-base \
--expose="$(git rev-parse --git-common-dir)" \
${SOURCES_PATH:+--share="$SOURCES_PATH"} \
${BASE_CACHE:+--share="$BASE_CACHE"} \
${SDK_PATH:+--share="$SDK_PATH"} \
--cores="$JOBS" \
--keep-failed \
--fallback \
--link-profile \
--root="$(profiledir_for_host "${HOST}")" \
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
-- env HOST="$host" \
DISTNAME="$DISTNAME" \
JOBS="$JOBS" \
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
${V:+V=1} \
${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
${BASE_CACHE:+BASE_CACHE="$BASE_CACHE"} \
${SDK_PATH:+SDK_PATH="$SDK_PATH"} \
DISTSRC="$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")" \
OUTDIR="$(OUTDIR_BASE=/outdir-base && outdir_for_host "$HOST")" \
DIST_ARCHIVE_BASE=/outdir-base/dist-archive \
bash -c "cd /digibyte && bash contrib/guix/libexec/build.sh"
)
done

83
contrib/guix/guix-clean Executable file
View file

@ -0,0 +1,83 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
# Source the common prelude, which:
# 1. Checks if we're at the top directory of the DigiByte Core repository
# 2. Defines a few common functions and variables
#
# shellcheck source=libexec/prelude.bash
source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
###################
## Sanity Checks ##
###################
################
# Required non-builtin commands should be invokable
################
check_tools cat mkdir make git guix
#############
## Clean ##
#############
# Usage: under_dir MAYBE_PARENT MAYBE_CHILD
#
# If MAYBE_CHILD is a subdirectory of MAYBE_PARENT, print the relative path
# from MAYBE_PARENT to MAYBE_CHILD. Otherwise, return 1 as the error code.
#
# NOTE: This does not perform any symlink-resolving or path canonicalization.
#
under_dir() {
local path_residue
path_residue="${2##${1}}"
if [ -z "$path_residue" ] || [ "$path_residue" = "$2" ]; then
return 1
else
echo "$path_residue"
fi
}
# Usage: dir_under_git_root MAYBE_CHILD
#
# If MAYBE_CHILD is under the current git repository and exists, print the
# relative path from the git repository's top-level directory to MAYBE_CHILD,
# otherwise, exit with an error code.
#
dir_under_git_root() {
local rv
rv="$(under_dir "$(git_root)" "$1")"
[ -n "$rv" ] && echo "$rv"
}
shopt -s nullglob
found_precious_dirs_files=( "${version_base_prefix}"*/"${var_base_basename}/precious_dirs" ) # This expands to an array of directories...
shopt -u nullglob
exclude_flags=()
for precious_dirs_file in "${found_precious_dirs_files[@]}"; do
# Make sure the precious directories (e.g. SOURCES_PATH, BASE_CACHE, SDK_PATH)
# are excluded from git-clean
echo "Found precious_dirs file: '${precious_dirs_file}'"
# Exclude the precious_dirs file itself
if dirs_file_exclude_fragment=$(dir_under_git_root "$(dirname "$precious_dirs_file")"); then
exclude_flags+=( --exclude="${dirs_file_exclude_fragment}/precious_dirs" )
fi
# Read each 'name=dir' pair from the precious_dirs file
while IFS='=' read -r name dir; do
# Add an exclusion flag if the precious directory is under the git root.
if under=$(dir_under_git_root "$dir"); then
echo "Avoiding ${name}: ${under}"
exclude_flags+=( --exclude="$under" )
fi
done < "$precious_dirs_file"
done
git clean -xdff "${exclude_flags[@]}"

392
contrib/guix/guix-codesign Executable file
View file

@ -0,0 +1,392 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
# Source the common prelude, which:
# 1. Checks if we're at the top directory of the DigiByte Core repository
# 2. Defines a few common functions and variables
#
# shellcheck source=libexec/prelude.bash
source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
###################
## SANITY CHECKS ##
###################
################
# Required non-builtin commands should be invocable
################
check_tools cat mkdir git guix
################
# Required env vars should be non-empty
################
cmd_usage() {
cat <<EOF
Synopsis:
env DETACHED_SIGS_REPO=<path/to/digibyte-detached-sigs> \\
./contrib/guix/guix-codesign
EOF
}
if [ -z "$DETACHED_SIGS_REPO" ]; then
cmd_usage
exit 1
fi
################
# GUIX_BUILD_OPTIONS should be empty
################
#
# GUIX_BUILD_OPTIONS is an environment variable recognized by guix commands that
# can perform builds. This seems like what we want instead of
# ADDITIONAL_GUIX_COMMON_FLAGS, but the value of GUIX_BUILD_OPTIONS is actually
# _appended_ to normal command-line options. Meaning that they will take
# precedence over the command-specific ADDITIONAL_GUIX_<CMD>_FLAGS.
#
# This seems like a poor user experience. Thus we check for GUIX_BUILD_OPTIONS's
# existence here and direct users of this script to use our (more flexible)
# custom environment variables.
if [ -n "$GUIX_BUILD_OPTIONS" ]; then
cat << EOF
Error: Environment variable GUIX_BUILD_OPTIONS is not empty:
'$GUIX_BUILD_OPTIONS'
Unfortunately this script is incompatible with GUIX_BUILD_OPTIONS, please unset
GUIX_BUILD_OPTIONS and use ADDITIONAL_GUIX_COMMON_FLAGS to set build options
across guix commands or ADDITIONAL_GUIX_<CMD>_FLAGS to set build options for a
specific guix command.
See contrib/guix/README.md for more details.
EOF
exit 1
fi
################
# The codesignature git worktree should not be dirty
################
if ! git -C "$DETACHED_SIGS_REPO" diff-index --quiet HEAD -- && [ -z "$FORCE_DIRTY_WORKTREE" ]; then
cat << EOF
ERR: The DETACHED CODESIGNATURE git worktree is dirty, which may lead to broken builds.
Aborting...
Hint: To make your git worktree clean, You may want to:
1. Commit your changes,
2. Stash your changes, or
3. Set the 'FORCE_DIRTY_WORKTREE' environment variable if you insist on
using a dirty worktree
EOF
exit 1
fi
################
# Build directories should not exist
################
# Default to building for all supported HOSTs (overridable by environment)
export HOSTS="${HOSTS:-x86_64-w64-mingw32 x86_64-apple-darwin arm64-apple-darwin}"
# Usage: distsrc_for_host HOST
#
# HOST: The current platform triple we're building for
#
distsrc_for_host() {
echo "${DISTSRC_BASE}/distsrc-${VERSION}-${1}-codesigned"
}
# Accumulate a list of build directories that already exist...
hosts_distsrc_exists=""
for host in $HOSTS; do
if [ -e "$(distsrc_for_host "$host")" ]; then
hosts_distsrc_exists+=" ${host}"
fi
done
if [ -n "$hosts_distsrc_exists" ]; then
# ...so that we can print them out nicely in an error message
cat << EOF
ERR: Build directories for this commit already exist for the following platform
triples you're attempting to build, probably because of previous builds.
Please remove, or otherwise deal with them prior to starting another build.
Aborting...
Hint: To blow everything away, you may want to use:
$ ./contrib/guix/guix-clean
Specifically, this will remove all files without an entry in the index,
excluding the SDK directory, the depends download cache, the depends built
packages cache, the garbage collector roots for Guix environments, and the
output directory.
EOF
for host in $hosts_distsrc_exists; do
echo " ${host} '$(distsrc_for_host "$host")'"
done
exit 1
else
mkdir -p "$DISTSRC_BASE"
fi
################
# Unsigned tarballs SHOULD exist
################
# Usage: outdir_for_host HOST SUFFIX
#
# HOST: The current platform triple we're building for
#
outdir_for_host() {
echo "${OUTDIR_BASE}/${1}${2:+-${2}}"
}
unsigned_tarball_for_host() {
case "$1" in
*mingw*)
echo "$(outdir_for_host "$1")/${DISTNAME}-win-unsigned.tar.gz"
;;
*darwin*)
echo "$(outdir_for_host "$1")/${DISTNAME}-osx-unsigned.tar.gz"
;;
*)
exit 1
;;
esac
}
# Accumulate a list of build directories that already exist...
hosts_unsigned_tarball_missing=""
for host in $HOSTS; do
if [ ! -e "$(unsigned_tarball_for_host "$host")" ]; then
hosts_unsigned_tarball_missing+=" ${host}"
fi
done
if [ -n "$hosts_unsigned_tarball_missing" ]; then
# ...so that we can print them out nicely in an error message
cat << EOF
ERR: Unsigned tarballs do not exist
...
EOF
for host in $hosts_unsigned_tarball_missing; do
echo " ${host} '$(unsigned_tarball_for_host "$host")'"
done
exit 1
fi
################
# Check that we can connect to the guix-daemon
################
cat << EOF
Checking that we can connect to the guix-daemon...
Hint: If this hangs, you may want to try turning your guix-daemon off and on
again.
EOF
if ! guix gc --list-failures > /dev/null; then
cat << EOF
ERR: Failed to connect to the guix-daemon, please ensure that one is running and
reachable.
EOF
exit 1
fi
# Developer note: we could use `guix repl` for this check and run:
#
# (import (guix store)) (close-connection (open-connection))
#
# However, the internal API is likely to change more than the CLI invocation
#########
# SETUP #
#########
# Determine the maximum number of jobs to run simultaneously (overridable by
# environment)
JOBS="${JOBS:-$(nproc)}"
# Determine the reference time used for determinism (overridable by environment)
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
# Execute "$@" in a pinned, possibly older version of Guix, for reproducibility
# across time.
time-machine() {
# shellcheck disable=SC2086
guix time-machine --url=https://git.savannah.gnu.org/git/guix.git \
--commit=aa34d4d28dfe25ba47d5800d05000fb7221788c0 \
--cores="$JOBS" \
--keep-failed \
--fallback \
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \
-- "$@"
}
# Make sure an output directory exists for our builds
OUTDIR_BASE="${OUTDIR_BASE:-${VERSION_BASE}/output}"
mkdir -p "$OUTDIR_BASE"
# Usage: profiledir_for_host HOST SUFFIX
#
# HOST: The current platform triple we're building for
#
profiledir_for_host() {
echo "${PROFILES_BASE}/${1}${2:+-${2}}"
}
#########
# BUILD #
#########
# Function to be called when codesigning for host ${1} and the user interrupts
# the codesign
int_trap() {
cat << EOF
** INT received while codesigning ${1}, you may want to clean up the relevant
work directories (e.g. distsrc-*) before recodesigning
Hint: To blow everything away, you may want to use:
$ ./contrib/guix/guix-clean
Specifically, this will remove all files without an entry in the index,
excluding the SDK directory, the depends download cache, the depends built
packages cache, the garbage collector roots for Guix environments, and the
output directory.
EOF
}
# Deterministically build DigiByte Core
# shellcheck disable=SC2153
for host in $HOSTS; do
# Display proper warning when the user interrupts the build
trap 'int_trap ${host}' INT
(
# Required for 'contrib/guix/manifest.scm' to output the right manifest
# for the particular $HOST we're building for
export HOST="$host"
# shellcheck disable=SC2030
cat << EOF
INFO: Codesigning ${VERSION:?not set} for platform triple ${HOST:?not set}:
...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set}
...from worktree directory: '${PWD}'
...bind-mounted in container to: '/digibyte'
...in build directory: '$(distsrc_for_host "$HOST")'
...bind-mounted in container to: '$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")'
...outputting in: '$(outdir_for_host "$HOST" codesigned)'
...bind-mounted in container to: '$(OUTDIR_BASE=/outdir-base && outdir_for_host "$HOST" codesigned)'
...using detached signatures in: '${DETACHED_SIGS_REPO:?not set}'
...bind-mounted in container to: '/detached-sigs'
EOF
# Run the build script 'contrib/guix/libexec/build.sh' in the build
# container specified by 'contrib/guix/manifest.scm'.
#
# Explanation of `guix environment` flags:
#
# --container run command within an isolated container
#
# Running in an isolated container minimizes build-time differences
# between machines and improves reproducibility
#
# --pure unset existing environment variables
#
# Same rationale as --container
#
# --no-cwd do not share current working directory with an
# isolated container
#
# When --container is specified, the default behavior is to share
# the current working directory with the isolated container at the
# same exact path (e.g. mapping '/home/satoshi/digibyte/' to
# '/home/satoshi/digibyte/'). This means that the $PWD inside the
# container becomes a source of irreproducibility. --no-cwd disables
# this behaviour.
#
# --share=SPEC for containers, share writable host file system
# according to SPEC
#
# --share="$PWD"=/digibyte
#
# maps our current working directory to /digibyte
# inside the isolated container, which we later cd
# into.
#
# While we don't want to map our current working directory to the
# same exact path (as this introduces irreproducibility), we do want
# it to be at a _fixed_ path _somewhere_ inside the isolated
# container so that we have something to build. '/digibyte' was
# chosen arbitrarily.
#
# ${SOURCES_PATH:+--share="$SOURCES_PATH"}
#
# make the downloaded depends sources path available
# inside the isolated container
#
# The isolated container has no network access as it's in a
# different network namespace from the main machine, so we have to
# make the downloaded depends sources available to it. The sources
# should have been downloaded prior to this invocation.
#
# ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"}
#
# fetch substitute from SUBSTITUTE_URLS if they are
# authorized
#
# Depending on the user's security model, it may be desirable to use
# substitutes (pre-built packages) from servers that the user trusts.
# Please read the README.md in the same directory as this file for
# more information.
#
# shellcheck disable=SC2086,SC2031
time-machine environment --manifest="${PWD}/contrib/guix/manifest.scm" \
--container \
--pure \
--no-cwd \
--share="$PWD"=/digibyte \
--share="$DISTSRC_BASE"=/distsrc-base \
--share="$OUTDIR_BASE"=/outdir-base \
--share="$DETACHED_SIGS_REPO"=/detached-sigs \
--expose="$(git rev-parse --git-common-dir)" \
--expose="$(git -C "$DETACHED_SIGS_REPO" rev-parse --git-common-dir)" \
${SOURCES_PATH:+--share="$SOURCES_PATH"} \
--cores="$JOBS" \
--keep-failed \
--fallback \
--link-profile \
--root="$(profiledir_for_host "${HOST}" codesigned)" \
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
-- env HOST="$host" \
DISTNAME="$DISTNAME" \
JOBS="$JOBS" \
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
${V:+V=1} \
${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
DISTSRC="$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")" \
OUTDIR="$(OUTDIR_BASE=/outdir-base && outdir_for_host "$HOST" codesigned)" \
DIST_ARCHIVE_BASE=/outdir-base/dist-archive \
DETACHED_SIGS_REPO=/detached-sigs \
UNSIGNED_TARBALL="$(OUTDIR_BASE=/outdir-base && unsigned_tarball_for_host "$HOST")" \
bash -c "cd /digibyte && bash contrib/guix/libexec/codesign.sh"
)
done

142
contrib/guix/guix-verify Executable file
View file

@ -0,0 +1,142 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
# Source the common prelude, which:
# 1. Checks if we're at the top directory of the DigiByte Core repository
# 2. Defines a few common functions and variables
#
# shellcheck source=libexec/prelude.bash
source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
###################
## Sanity Checks ##
###################
################
# Required non-builtin commands should be invokable
################
check_tools cat diff gpg
################
# Required env vars should be non-empty
################
cmd_usage() {
cat <<EOF
Synopsis:
env GUIX_SIGS_REPO=<path/to/guix.sigs> ./contrib/guix/guix-verify
EOF
}
if [ -z "$GUIX_SIGS_REPO" ]; then
cmd_usage
exit 1
fi
################
# GUIX_SIGS_REPO should exist as a directory
################
if [ ! -d "$GUIX_SIGS_REPO" ]; then
cat << EOF
ERR: The specified GUIX_SIGS_REPO is not an existent directory:
'$GUIX_SIGS_REPO'
Hint: Please clone the guix.sigs repository and point to it with the
GUIX_SIGS_REPO environment variable.
EOF
cmd_usage
exit 1
fi
##############
## Verify ##
##############
OUTSIGDIR_BASE="${GUIX_SIGS_REPO}/${VERSION}"
echo "Looking for signature directories in '${OUTSIGDIR_BASE}'"
echo ""
# Usage: verify compare_manifest current_manifest
verify() {
local compare_manifest="$1"
local current_manifest="$2"
if ! gpg --quiet --batch --verify "$current_manifest".asc "$current_manifest" 1>&2; then
echo "ERR: Failed to verify GPG signature in '${current_manifest}'"
echo ""
echo "Hint: Either the signature is invalid or the public key is missing"
echo ""
elif ! diff --report-identical "$compare_manifest" "$current_manifest" 1>&2; then
echo "ERR: The SHA256SUMS attestation in these two directories differ:"
echo " '${compare_manifest}'"
echo " '${current_manifest}'"
echo ""
else
echo "Verified: '${current_manifest}'"
echo ""
fi
}
shopt -s nullglob
all_noncodesigned=( "$OUTSIGDIR_BASE"/*/noncodesigned.SHA256SUMS )
shopt -u nullglob
echo "--------------------"
echo ""
if (( ${#all_noncodesigned[@]} )); then
compare_noncodesigned="${all_noncodesigned[0]}"
for current_manifest in "${all_noncodesigned[@]}"; do
verify "$compare_noncodesigned" "$current_manifest"
done
echo "DONE: Checking output signatures for noncodesigned.SHA256SUMS"
echo ""
else
echo "WARN: No signature directories with noncodesigned.SHA256SUMS found"
echo ""
fi
shopt -s nullglob
all_all=( "$OUTSIGDIR_BASE"/*/all.SHA256SUMS )
shopt -u nullglob
echo "--------------------"
echo ""
if (( ${#all_all[@]} )); then
compare_all="${all_all[0]}"
for current_manifest in "${all_all[@]}"; do
verify "$compare_all" "$current_manifest"
done
# Sanity check: there should be no entries that exist in
# noncodesigned.SHA256SUMS that doesn't exist in all.SHA256SUMS
if [[ "$(comm -23 <(sort "$compare_noncodesigned") <(sort "$compare_all") | wc -c)" -ne 0 ]]; then
echo "ERR: There are unique lines in noncodesigned.SHA256SUMS which"
echo " do not exist in all.SHA256SUMS, something went very wrong."
exit 1
fi
echo "DONE: Checking output signatures for all.SHA256SUMS"
echo ""
else
echo "WARN: No signature directories with all.SHA256SUMS found"
echo ""
fi
echo "===================="
echo ""
if (( ${#all_noncodesigned[@]} + ${#all_all[@]} == 0 )); then
echo "ERR: Unable to perform any verifications as no signature directories"
echo " were found"
echo ""
exit 1
fi

449
contrib/guix/libexec/build.sh Executable file
View file

@ -0,0 +1,449 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
export TZ=UTC
# Although Guix _does_ set umask when building its own packages (in our case,
# this is all packages in manifest.scm), it does not set it for `guix
# environment`. It does make sense for at least `guix environment --container`
# to set umask, so if that change gets merged upstream and we bump the
# time-machine to a commit which includes the aforementioned change, we can
# remove this line.
#
# This line should be placed before any commands which creates files.
umask 0022
if [ -n "$V" ]; then
# Print both unexpanded (-v) and expanded (-x) forms of commands as they are
# read from this file.
set -vx
# Set VERBOSE for CMake-based builds
export VERBOSE="$V"
fi
# Check that required environment variables are set
cat << EOF
Required environment variables as seen inside the container:
DIST_ARCHIVE_BASE: ${DIST_ARCHIVE_BASE:?not set}
DISTNAME: ${DISTNAME:?not set}
HOST: ${HOST:?not set}
SOURCE_DATE_EPOCH: ${SOURCE_DATE_EPOCH:?not set}
JOBS: ${JOBS:?not set}
DISTSRC: ${DISTSRC:?not set}
OUTDIR: ${OUTDIR:?not set}
EOF
ACTUAL_OUTDIR="${OUTDIR}"
OUTDIR="${DISTSRC}/output"
#####################
# Environment Setup #
#####################
# The depends folder also serves as a base-prefix for depends packages for
# $HOSTs after successfully building.
BASEPREFIX="${PWD}/depends"
# Given a package name and an output name, return the path of that output in our
# current guix environment
store_path() {
grep --extended-regexp "/[^-]{32}-${1}-[^-]+${2:+-${2}}" "${GUIX_ENVIRONMENT}/manifest" \
| head --lines=1 \
| sed --expression='s|^[[:space:]]*"||' \
--expression='s|"[[:space:]]*$||'
}
# Set environment variables to point the NATIVE toolchain to the right
# includes/libs
NATIVE_GCC="$(store_path gcc-toolchain)"
NATIVE_GCC_STATIC="$(store_path gcc-toolchain static)"
unset LIBRARY_PATH
unset CPATH
unset C_INCLUDE_PATH
unset CPLUS_INCLUDE_PATH
unset OBJC_INCLUDE_PATH
unset OBJCPLUS_INCLUDE_PATH
export LIBRARY_PATH="${NATIVE_GCC}/lib:${NATIVE_GCC}/lib64:${NATIVE_GCC_STATIC}/lib:${NATIVE_GCC_STATIC}/lib64"
export C_INCLUDE_PATH="${NATIVE_GCC}/include"
export CPLUS_INCLUDE_PATH="${NATIVE_GCC}/include/c++:${NATIVE_GCC}/include"
export OBJC_INCLUDE_PATH="${NATIVE_GCC}/include"
export OBJCPLUS_INCLUDE_PATH="${NATIVE_GCC}/include/c++:${NATIVE_GCC}/include"
prepend_to_search_env_var() {
export "${1}=${2}${!1:+:}${!1}"
}
# Set environment variables to point the CROSS toolchain to the right
# includes/libs for $HOST
case "$HOST" in
*mingw*)
# Determine output paths to use in CROSS_* environment variables
CROSS_GLIBC="$(store_path "mingw-w64-x86_64-winpthreads")"
CROSS_GCC="$(store_path "gcc-cross-${HOST}")"
CROSS_GCC_LIB_STORE="$(store_path "gcc-cross-${HOST}" lib)"
CROSS_GCC_LIBS=( "${CROSS_GCC_LIB_STORE}/lib/gcc/${HOST}"/* ) # This expands to an array of directories...
CROSS_GCC_LIB="${CROSS_GCC_LIBS[0]}" # ...we just want the first one (there should only be one)
# The search path ordering is generally:
# 1. gcc-related search paths
# 2. libc-related search paths
# 2. kernel-header-related search paths (not applicable to mingw-w64 hosts)
export CROSS_C_INCLUDE_PATH="${CROSS_GCC_LIB}/include:${CROSS_GCC_LIB}/include-fixed:${CROSS_GLIBC}/include"
export CROSS_CPLUS_INCLUDE_PATH="${CROSS_GCC}/include/c++:${CROSS_GCC}/include/c++/${HOST}:${CROSS_GCC}/include/c++/backward:${CROSS_C_INCLUDE_PATH}"
export CROSS_LIBRARY_PATH="${CROSS_GCC_LIB_STORE}/lib:${CROSS_GCC}/${HOST}/lib:${CROSS_GCC_LIB}:${CROSS_GLIBC}/lib"
;;
*darwin*)
# The CROSS toolchain for darwin uses the SDK and ignores environment variables.
# See depends/hosts/darwin.mk for more details.
;;
*linux*)
CROSS_GLIBC="$(store_path "glibc-cross-${HOST}")"
CROSS_GLIBC_STATIC="$(store_path "glibc-cross-${HOST}" static)"
CROSS_KERNEL="$(store_path "linux-libre-headers-cross-${HOST}")"
CROSS_GCC="$(store_path "gcc-cross-${HOST}")"
CROSS_GCC_LIB_STORE="$(store_path "gcc-cross-${HOST}" lib)"
CROSS_GCC_LIBS=( "${CROSS_GCC_LIB_STORE}/lib/gcc/${HOST}"/* ) # This expands to an array of directories...
CROSS_GCC_LIB="${CROSS_GCC_LIBS[0]}" # ...we just want the first one (there should only be one)
export CROSS_C_INCLUDE_PATH="${CROSS_GCC_LIB}/include:${CROSS_GCC_LIB}/include-fixed:${CROSS_GLIBC}/include:${CROSS_KERNEL}/include"
export CROSS_CPLUS_INCLUDE_PATH="${CROSS_GCC}/include/c++:${CROSS_GCC}/include/c++/${HOST}:${CROSS_GCC}/include/c++/backward:${CROSS_C_INCLUDE_PATH}"
export CROSS_LIBRARY_PATH="${CROSS_GCC_LIB_STORE}/lib:${CROSS_GCC}/${HOST}/lib:${CROSS_GCC_LIB}:${CROSS_GLIBC}/lib:${CROSS_GLIBC_STATIC}/lib"
;;
*)
exit 1 ;;
esac
# Sanity check CROSS_*_PATH directories
IFS=':' read -ra PATHS <<< "${CROSS_C_INCLUDE_PATH}:${CROSS_CPLUS_INCLUDE_PATH}:${CROSS_LIBRARY_PATH}"
for p in "${PATHS[@]}"; do
if [ -n "$p" ] && [ ! -d "$p" ]; then
echo "'$p' doesn't exist or isn't a directory... Aborting..."
exit 1
fi
done
# Disable Guix ld auto-rpath behavior
case "$HOST" in
*darwin*)
# The auto-rpath behavior is necessary for darwin builds as some native
# tools built by depends refer to and depend on Guix-built native
# libraries
#
# After the native packages in depends are built, the ld wrapper should
# no longer affect our build, as clang would instead reach for
# x86_64-apple-darwin18-ld from cctools
;;
*) export GUIX_LD_WRAPPER_DISABLE_RPATH=yes ;;
esac
# Make /usr/bin if it doesn't exist
[ -e /usr/bin ] || mkdir -p /usr/bin
# Symlink file and env to a conventional path
[ -e /usr/bin/file ] || ln -s --no-dereference "$(command -v file)" /usr/bin/file
[ -e /usr/bin/env ] || ln -s --no-dereference "$(command -v env)" /usr/bin/env
# Determine the correct value for -Wl,--dynamic-linker for the current $HOST
case "$HOST" in
*linux*)
glibc_dynamic_linker=$(
case "$HOST" in
i686-linux-gnu) echo /lib/ld-linux.so.2 ;;
x86_64-linux-gnu) echo /lib64/ld-linux-x86-64.so.2 ;;
arm-linux-gnueabihf) echo /lib/ld-linux-armhf.so.3 ;;
aarch64-linux-gnu) echo /lib/ld-linux-aarch64.so.1 ;;
riscv64-linux-gnu) echo /lib/ld-linux-riscv64-lp64d.so.1 ;;
powerpc64-linux-gnu) echo /lib64/ld64.so.1;;
powerpc64le-linux-gnu) echo /lib64/ld64.so.2;;
*) exit 1 ;;
esac
)
;;
esac
# Environment variables for determinism
export TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --mtime='@${SOURCE_DATE_EPOCH}' --sort=name"
export TZ="UTC"
case "$HOST" in
*darwin*)
# cctools AR, unlike GNU binutils AR, does not have a deterministic mode
# or a configure flag to enable determinism by default, it only
# understands if this env-var is set or not. See:
#
# https://github.com/tpoechtrager/cctools-port/blob/55562e4073dea0fbfd0b20e0bf69ffe6390c7f97/cctools/ar/archive.c#L334
export ZERO_AR_DATE=yes
;;
esac
####################
# Depends Building #
####################
# Build the depends tree, overriding variables that assume multilib gcc
make -C depends --jobs="$JOBS" HOST="$HOST" \
${V:+V=1} \
${SOURCES_PATH+SOURCES_PATH="$SOURCES_PATH"} \
${BASE_CACHE+BASE_CACHE="$BASE_CACHE"} \
${SDK_PATH+SDK_PATH="$SDK_PATH"} \
i686_linux_CC=i686-linux-gnu-gcc \
i686_linux_CXX=i686-linux-gnu-g++ \
i686_linux_AR=i686-linux-gnu-ar \
i686_linux_RANLIB=i686-linux-gnu-ranlib \
i686_linux_NM=i686-linux-gnu-nm \
i686_linux_STRIP=i686-linux-gnu-strip \
x86_64_linux_CC=x86_64-linux-gnu-gcc \
x86_64_linux_CXX=x86_64-linux-gnu-g++ \
x86_64_linux_AR=x86_64-linux-gnu-ar \
x86_64_linux_RANLIB=x86_64-linux-gnu-ranlib \
x86_64_linux_NM=x86_64-linux-gnu-nm \
x86_64_linux_STRIP=x86_64-linux-gnu-strip \
qt_config_opts_i686_linux='-platform linux-g++ -xplatform digibyte-linux-g++' \
qt_config_opts_x86_64_linux='-platform linux-g++ -xplatform digibyte-linux-g++' \
FORCE_USE_SYSTEM_CLANG=1
###########################
# Source Tarball Building #
###########################
GIT_ARCHIVE="${DIST_ARCHIVE_BASE}/${DISTNAME}.tar.gz"
# Create the source tarball if not already there
if [ ! -e "$GIT_ARCHIVE" ]; then
mkdir -p "$(dirname "$GIT_ARCHIVE")"
touch "${DIST_ARCHIVE_BASE}"/SKIPATTEST.TAG
git archive --prefix="${DISTNAME}/" --output="$GIT_ARCHIVE" HEAD
fi
mkdir -p "$OUTDIR"
###########################
# Binary Tarball Building #
###########################
# CONFIGFLAGS
CONFIGFLAGS="--enable-reduce-exports --disable-bench --disable-gui-tests --disable-fuzz-binary"
case "$HOST" in
*linux*) CONFIGFLAGS+=" --disable-threadlocal" ;;
esac
# CFLAGS
HOST_CFLAGS="-O2 -g"
case "$HOST" in
*linux*) HOST_CFLAGS+=" -ffile-prefix-map=${PWD}=." ;;
*mingw*) HOST_CFLAGS+=" -fno-ident" ;;
*darwin*) unset HOST_CFLAGS ;;
esac
# CXXFLAGS
HOST_CXXFLAGS="$HOST_CFLAGS"
case "$HOST" in
arm-linux-gnueabihf) HOST_CXXFLAGS="${HOST_CXXFLAGS} -Wno-psabi" ;;
esac
# LDFLAGS
case "$HOST" in
*linux*) HOST_LDFLAGS="-Wl,--as-needed -Wl,--dynamic-linker=$glibc_dynamic_linker -static-libstdc++ -Wl,-O2" ;;
*mingw*) HOST_LDFLAGS="-Wl,--no-insert-timestamp" ;;
esac
# Using --no-tls-get-addr-optimize retains compatibility with glibc 2.17, by
# avoiding a PowerPC64 optimisation available in glibc 2.22 and later.
# https://sourceware.org/binutils/docs-2.35/ld/PowerPC64-ELF64.html
case "$HOST" in
*powerpc64*) HOST_LDFLAGS="${HOST_LDFLAGS} -Wl,--no-tls-get-addr-optimize" ;;
esac
case "$HOST" in
powerpc64-linux-*|riscv64-linux-*) HOST_LDFLAGS="${HOST_LDFLAGS} -Wl,-z,noexecstack" ;;
esac
# Make $HOST-specific native binaries from depends available in $PATH
export PATH="${BASEPREFIX}/${HOST}/native/bin:${PATH}"
mkdir -p "$DISTSRC"
(
cd "$DISTSRC"
# Extract the source tarball
tar --strip-components=1 -xf "${GIT_ARCHIVE}"
./autogen.sh
# Configure this DISTSRC for $HOST
# shellcheck disable=SC2086
env CONFIG_SITE="${BASEPREFIX}/${HOST}/share/config.site" \
./configure --prefix=/ \
--disable-ccache \
--disable-maintainer-mode \
--disable-dependency-tracking \
${CONFIGFLAGS} \
${HOST_CFLAGS:+CFLAGS="${HOST_CFLAGS}"} \
${HOST_CXXFLAGS:+CXXFLAGS="${HOST_CXXFLAGS}"} \
${HOST_LDFLAGS:+LDFLAGS="${HOST_LDFLAGS}"}
sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool
# Build DigiByte Core
make --jobs="$JOBS" ${V:+V=1}
# Check that symbol/security checks tools are sane.
make test-security-check ${V:+V=1}
# Perform basic security checks on a series of executables.
make -C src --jobs=1 check-security ${V:+V=1}
# Check that executables only contain allowed version symbols.
make -C src --jobs=1 check-symbols ${V:+V=1}
mkdir -p "$OUTDIR"
# Make the os-specific installers
case "$HOST" in
*mingw*)
make deploy ${V:+V=1} DIGIBYTE_WIN_INSTALLER="${OUTDIR}/${DISTNAME}-win64-setup-unsigned.exe"
;;
esac
# Setup the directory where our DigiByte Core build for HOST will be
# installed. This directory will also later serve as the input for our
# binary tarballs.
INSTALLPATH="${PWD}/installed/${DISTNAME}"
mkdir -p "${INSTALLPATH}"
# Install built DigiByte Core to $INSTALLPATH
case "$HOST" in
*darwin*)
make install-strip DESTDIR="${INSTALLPATH}" ${V:+V=1}
;;
*)
make install DESTDIR="${INSTALLPATH}" ${V:+V=1}
;;
esac
case "$HOST" in
*darwin*)
make osx_volname ${V:+V=1}
make deploydir ${V:+V=1}
mkdir -p "unsigned-app-${HOST}"
cp --target-directory="unsigned-app-${HOST}" \
osx_volname \
contrib/macdeploy/detached-sig-create.sh
mv --target-directory="unsigned-app-${HOST}" dist
(
cd "unsigned-app-${HOST}"
find . -print0 \
| sort --zero-terminated \
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
| gzip -9n > "${OUTDIR}/${DISTNAME}-osx-unsigned.tar.gz" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-osx-unsigned.tar.gz" && exit 1 )
)
make deploy ${V:+V=1} OSX_DMG="${OUTDIR}/${DISTNAME}-osx-unsigned.dmg"
;;
esac
(
cd installed
case "$HOST" in
*mingw*)
mv --target-directory="$DISTNAME"/lib/ "$DISTNAME"/bin/*.dll
;;
esac
# Prune libtool and object archives
find . -name "lib*.la" -delete
find . -name "lib*.a" -delete
# Prune pkg-config files
rm -rf "${DISTNAME}/lib/pkgconfig"
case "$HOST" in
*darwin*) ;;
*)
# Split binaries and libraries from their debug symbols
{
find "${DISTNAME}/bin" -type f -executable -print0
find "${DISTNAME}/lib" -type f -print0
} | xargs -0 -n1 -P"$JOBS" -I{} "${DISTSRC}/contrib/devtools/split-debug.sh" {} {} {}.dbg
;;
esac
case "$HOST" in
*mingw*)
cp "${DISTSRC}/doc/README_windows.txt" "${DISTNAME}/readme.txt"
;;
*linux*)
cp "${DISTSRC}/README.md" "${DISTNAME}/"
;;
esac
# Finally, deterministically produce {non-,}debug binary tarballs ready
# for release
case "$HOST" in
*mingw*)
find "${DISTNAME}" -not -name "*.dbg" -print0 \
| xargs -0r touch --no-dereference --date="@${SOURCE_DATE_EPOCH}"
find "${DISTNAME}" -not -name "*.dbg" \
| sort \
| zip -X@ "${OUTDIR}/${DISTNAME}-${HOST//x86_64-w64-mingw32/win64}.zip" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST//x86_64-w64-mingw32/win64}.zip" && exit 1 )
find "${DISTNAME}" -name "*.dbg" -print0 \
| xargs -0r touch --no-dereference --date="@${SOURCE_DATE_EPOCH}"
find "${DISTNAME}" -name "*.dbg" \
| sort \
| zip -X@ "${OUTDIR}/${DISTNAME}-${HOST//x86_64-w64-mingw32/win64}-debug.zip" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST//x86_64-w64-mingw32/win64}-debug.zip" && exit 1 )
;;
*linux*)
find "${DISTNAME}" -not -name "*.dbg" -print0 \
| sort --zero-terminated \
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
| gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}.tar.gz" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}.tar.gz" && exit 1 )
find "${DISTNAME}" -name "*.dbg" -print0 \
| sort --zero-terminated \
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
| gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}-debug.tar.gz" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}-debug.tar.gz" && exit 1 )
;;
*darwin*)
find "${DISTNAME}" -print0 \
| sort --zero-terminated \
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
| gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST//x86_64-apple-darwin18/osx64}.tar.gz" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST//x86_64-apple-darwin18/osx64}.tar.gz" && exit 1 )
;;
esac
) # $DISTSRC/installed
case "$HOST" in
*mingw*)
cp -rf --target-directory=. contrib/windeploy
(
cd ./windeploy
mkdir -p unsigned
cp --target-directory=unsigned/ "${OUTDIR}/${DISTNAME}-win64-setup-unsigned.exe"
find . -print0 \
| sort --zero-terminated \
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
| gzip -9n > "${OUTDIR}/${DISTNAME}-win-unsigned.tar.gz" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-win-unsigned.tar.gz" && exit 1 )
)
;;
esac
) # $DISTSRC
rm -rf "$ACTUAL_OUTDIR"
mv --no-target-directory "$OUTDIR" "$ACTUAL_OUTDIR" \
|| ( rm -rf "$ACTUAL_OUTDIR" && exit 1 )
(
cd /outdir-base
{
echo "$GIT_ARCHIVE"
find "$ACTUAL_OUTDIR" -type f
} | xargs realpath --relative-base="$PWD" \
| xargs sha256sum \
| sort -k2 \
| sponge "$ACTUAL_OUTDIR"/SHA256SUMS.part
)

110
contrib/guix/libexec/codesign.sh Executable file
View file

@ -0,0 +1,110 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
export TZ=UTC
# Although Guix _does_ set umask when building its own packages (in our case,
# this is all packages in manifest.scm), it does not set it for `guix
# environment`. It does make sense for at least `guix environment --container`
# to set umask, so if that change gets merged upstream and we bump the
# time-machine to a commit which includes the aforementioned change, we can
# remove this line.
#
# This line should be placed before any commands which creates files.
umask 0022
if [ -n "$V" ]; then
# Print both unexpanded (-v) and expanded (-x) forms of commands as they are
# read from this file.
set -vx
# Set VERBOSE for CMake-based builds
export VERBOSE="$V"
fi
# Check that required environment variables are set
cat << EOF
Required environment variables as seen inside the container:
UNSIGNED_TARBALL: ${UNSIGNED_TARBALL:?not set}
DETACHED_SIGS_REPO: ${DETACHED_SIGS_REPO:?not set}
DIST_ARCHIVE_BASE: ${DIST_ARCHIVE_BASE:?not set}
DISTNAME: ${DISTNAME:?not set}
HOST: ${HOST:?not set}
SOURCE_DATE_EPOCH: ${SOURCE_DATE_EPOCH:?not set}
DISTSRC: ${DISTSRC:?not set}
OUTDIR: ${OUTDIR:?not set}
EOF
ACTUAL_OUTDIR="${OUTDIR}"
OUTDIR="${DISTSRC}/output"
git_head_version() {
local recent_tag
if recent_tag="$(git -C "$1" describe --exact-match HEAD 2> /dev/null)"; then
echo "${recent_tag#v}"
else
git -C "$1" rev-parse --short=12 HEAD
fi
}
CODESIGNATURE_GIT_ARCHIVE="${DIST_ARCHIVE_BASE}/${DISTNAME}-codesignatures-$(git_head_version "$DETACHED_SIGS_REPO").tar.gz"
# Create the codesignature tarball if not already there
if [ ! -e "$CODESIGNATURE_GIT_ARCHIVE" ]; then
mkdir -p "$(dirname "$CODESIGNATURE_GIT_ARCHIVE")"
git -C "$DETACHED_SIGS_REPO" archive --output="$CODESIGNATURE_GIT_ARCHIVE" HEAD
fi
mkdir -p "$OUTDIR"
mkdir -p "$DISTSRC"
(
cd "$DISTSRC"
tar -xf "$UNSIGNED_TARBALL"
mkdir -p codesignatures
tar -C codesignatures -xf "$CODESIGNATURE_GIT_ARCHIVE"
case "$HOST" in
*mingw*)
find "$PWD" -name "*-unsigned.exe" | while read -r infile; do
infile_base="$(basename "$infile")"
# Codesigned *-unsigned.exe and output to OUTDIR
osslsigncode attach-signature \
-in "$infile" \
-out "${OUTDIR}/${infile_base/-unsigned}" \
-sigin codesignatures/win/"$infile_base".pem
done
;;
*darwin*)
# Apply detached codesignatures to dist/ (in-place)
signapple apply dist/DigiByte-Qt.app codesignatures/osx/dist
# Make a DMG from dist/
xorrisofs -D -l -V "$(< osx_volname)" -no-pad -r -dir-mode 0755 \
-o "${OUTDIR}/${DISTNAME}-${HOST}.dmg" \
dist \
-- -volume_date all_file_dates ="$SOURCE_DATE_EPOCH"
;;
*)
exit 1
;;
esac
) # $DISTSRC
rm -rf "$ACTUAL_OUTDIR"
mv --no-target-directory "$OUTDIR" "$ACTUAL_OUTDIR" \
|| ( rm -rf "$ACTUAL_OUTDIR" && exit 1 )
(
cd /outdir-base
{
echo "$UNSIGNED_TARBALL"
echo "$CODESIGNATURE_GIT_ARCHIVE"
find "$ACTUAL_OUTDIR" -type f
} | xargs realpath --relative-base="$PWD" \
| xargs sha256sum \
| sort -k2 \
| sponge "$ACTUAL_OUTDIR"/SHA256SUMS.part
)

66
contrib/guix/libexec/prelude.bash Normal file
View file

@ -0,0 +1,66 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
# shellcheck source=../../shell/realpath.bash
source contrib/shell/realpath.bash
# shellcheck source=../../shell/git-utils.bash
source contrib/shell/git-utils.bash
################
# Required non-builtin commands should be invocable
################
check_tools() {
for cmd in "$@"; do
if ! command -v "$cmd" > /dev/null 2>&1; then
echo "ERR: This script requires that '$cmd' is installed and available in your \$PATH"
exit 1
fi
done
}
check_tools cat env readlink dirname basename git
################
# We should be at the top directory of the repository
################
same_dir() {
local resolved1 resolved2
resolved1="$(bash_realpath "${1}")"
resolved2="$(bash_realpath "${2}")"
[ "$resolved1" = "$resolved2" ]
}
if ! same_dir "${PWD}" "$(git_root)"; then
cat << EOF
ERR: This script must be invoked from the top level of the git repository
Hint: This may look something like:
env FOO=BAR ./contrib/guix/guix-<blah>
EOF
exit 1
fi
################
# Set common variables
################
VERSION="${VERSION:-$(git_head_version)}"
DISTNAME="${DISTNAME:-digibyte-${VERSION}}"
version_base_prefix="${PWD}/guix-build-"
VERSION_BASE="${version_base_prefix}${VERSION}" # TOP
DISTSRC_BASE="${DISTSRC_BASE:-${VERSION_BASE}}"
OUTDIR_BASE="${OUTDIR_BASE:-${VERSION_BASE}/output}"
var_base_basename="var"
VAR_BASE="${VAR_BASE:-${VERSION_BASE}/${var_base_basename}}"
profiles_base_basename="profiles"
PROFILES_BASE="${PROFILES_BASE:-${VAR_BASE}/${profiles_base_basename}}"

660
contrib/guix/manifest.scm Normal file
View file

@ -0,0 +1,660 @@
(use-modules (gnu)
(gnu packages)
(gnu packages autotools)
(gnu packages base)
(gnu packages bash)
(gnu packages bison)
(gnu packages certs)
(gnu packages cdrom)
(gnu packages check)
(gnu packages cmake)
(gnu packages commencement)
(gnu packages compression)
(gnu packages cross-base)
(gnu packages curl)
(gnu packages file)
(gnu packages gawk)
(gnu packages gcc)
(gnu packages gnome)
(gnu packages image)
(gnu packages imagemagick)
(gnu packages installers)
(gnu packages linux)
(gnu packages llvm)
(gnu packages mingw)
(gnu packages moreutils)
(gnu packages perl)
(gnu packages pkg-config)
(gnu packages python)
(gnu packages python-web)
(gnu packages shells)
(gnu packages tls)
(gnu packages version-control)
(guix build-system font)
(guix build-system gnu)
(guix build-system python)
(guix build-system trivial)
(guix download)
(guix gexp)
(guix git-download)
((guix licenses) #:prefix license:)
(guix packages)
(guix profiles)
(guix utils))
(define-syntax-rule (search-our-patches file-name ...)
"Return the list of absolute file names corresponding to each
FILE-NAME found in ./patches relative to the current file."
(parameterize
((%patch-path (list (string-append (dirname (current-filename)) "/patches"))))
(list (search-patch file-name) ...)))
(define (make-ssp-fixed-gcc xgcc)
"Given a XGCC package, return a modified package that uses the SSP function
from glibc instead of from libssp.so. Our `symbol-check' script will complain if
we link against libssp.so, and thus will ensure that this works properly.
Taken from:
http://www.linuxfromscratch.org/hlfs/view/development/chapter05/gcc-pass1.html"
(package
(inherit xgcc)
(arguments
(substitute-keyword-arguments (package-arguments xgcc)
((#:make-flags flags)
`(cons "gcc_cv_libc_provides_ssp=yes" ,flags))))))
(define (make-gcc-rpath-link xgcc)
"Given a XGCC package, return a modified package that replace each instance of
-rpath in the default system spec that's inserted by Guix with -rpath-link"
(package
(inherit xgcc)
(arguments
(substitute-keyword-arguments (package-arguments xgcc)
((#:phases phases)
`(modify-phases ,phases
(add-after 'pre-configure 'replace-rpath-with-rpath-link
(lambda _
(substitute* (cons "gcc/config/rs6000/sysv4.h"
(find-files "gcc/config"
"^gnu-user.*\\.h$"))
(("-rpath=") "-rpath-link="))
#t))))))))
(define (make-binutils-with-mingw-w64-disable-flags xbinutils)
(package-with-extra-patches xbinutils
(search-our-patches "binutils-mingw-w64-disable-flags.patch")))
(define (make-cross-toolchain target
base-gcc-for-libc
base-kernel-headers
base-libc
base-gcc)
"Create a cross-compilation toolchain package for TARGET"
(let* ((xbinutils (cross-binutils target))
;; 1. Build a cross-compiling gcc without targeting any libc, derived
;; from BASE-GCC-FOR-LIBC
(xgcc-sans-libc (cross-gcc target
#:xgcc base-gcc-for-libc
#:xbinutils xbinutils))
;; 2. Build cross-compiled kernel headers with XGCC-SANS-LIBC, derived
;; from BASE-KERNEL-HEADERS
(xkernel (cross-kernel-headers target
base-kernel-headers
xgcc-sans-libc
xbinutils))
;; 3. Build a cross-compiled libc with XGCC-SANS-LIBC and XKERNEL,
;; derived from BASE-LIBC
(xlibc (cross-libc target
base-libc
xgcc-sans-libc
xbinutils
xkernel))
;; 4. Build a cross-compiling gcc targeting XLIBC, derived from
;; BASE-GCC
(xgcc (cross-gcc target
#:xgcc base-gcc
#:xbinutils xbinutils
#:libc xlibc)))
;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
;; XGCC
(package
(name (string-append target "-toolchain"))
(version (package-version xgcc))
(source #f)
(build-system trivial-build-system)
(arguments '(#:builder (begin (mkdir %output) #t)))
(propagated-inputs
`(("binutils" ,xbinutils)
("libc" ,xlibc)
("libc:static" ,xlibc "static")
("gcc" ,xgcc)
("gcc-lib" ,xgcc "lib")))
(synopsis (string-append "Complete GCC tool chain for " target))
(description (string-append "This package provides a complete GCC tool
chain for " target " development."))
(home-page (package-home-page xgcc))
(license (package-license xgcc)))))
(define base-gcc
(package-with-extra-patches gcc-8
(search-our-patches "gcc-8-sort-libtool-find-output.patch")))
;; Building glibc with stack smashing protector first landed in glibc 2.25, use
;; this function to disable for older glibcs
;;
;; From glibc 2.25 changelog:
;;
;; * Most of glibc can now be built with the stack smashing protector enabled.
;; It is recommended to build glibc with --enable-stack-protector=strong.
;; Implemented by Nick Alcock (Oracle).
(define (make-glibc-without-ssp xglibc)
(package-with-extra-configure-variable
(package-with-extra-configure-variable
xglibc "libc_cv_ssp" "no")
"libc_cv_ssp_strong" "no"))
(define* (make-digibyte-cross-toolchain target
#:key
(base-gcc-for-libc gcc-7)
(base-kernel-headers linux-libre-headers-4.9)
(base-libc (make-glibc-without-ssp glibc-2.24))
(base-gcc (make-gcc-rpath-link base-gcc)))
"Convenience wrapper around MAKE-CROSS-TOOLCHAIN with default values
desirable for building DigiByte Core release binaries."
(make-cross-toolchain target
base-gcc-for-libc
base-kernel-headers
base-libc
base-gcc))
(define (make-gcc-with-pthreads gcc)
(package-with-extra-configure-variable gcc "--enable-threads" "posix"))
(define (make-mingw-pthreads-cross-toolchain target)
"Create a cross-compilation toolchain package for TARGET"
(let* ((xbinutils (make-binutils-with-mingw-w64-disable-flags (cross-binutils target)))
(pthreads-xlibc mingw-w64-x86_64-winpthreads)
(pthreads-xgcc (make-gcc-with-pthreads
(cross-gcc target
#:xgcc (make-ssp-fixed-gcc base-gcc)
#:xbinutils xbinutils
#:libc pthreads-xlibc))))
;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
;; XGCC
(package
(name (string-append target "-posix-toolchain"))
(version (package-version pthreads-xgcc))
(source #f)
(build-system trivial-build-system)
(arguments '(#:builder (begin (mkdir %output) #t)))
(propagated-inputs
`(("binutils" ,xbinutils)
("libc" ,pthreads-xlibc)
("gcc" ,pthreads-xgcc)
("gcc-lib" ,pthreads-xgcc "lib")))
(synopsis (string-append "Complete GCC tool chain for " target))
(description (string-append "This package provides a complete GCC tool
chain for " target " development."))
(home-page (package-home-page pthreads-xgcc))
(license (package-license pthreads-xgcc)))))
(define (make-nsis-with-sde-support base-nsis)
(package-with-extra-patches base-nsis
(search-our-patches "nsis-SConstruct-sde-support.patch")))
(define-public font-tuffy
(package
(name "font-tuffy")
(version "20120614")
(source
(origin
(method url-fetch)
(uri (string-append "http://tulrich.com/fonts/tuffy-" version ".tar.gz"))
(file-name (string-append name "-" version ".tar.gz"))
(sha256
(base32
"02vf72bgrp30vrbfhxjw82s115z27dwfgnmmzfb0n9wfhxxfpyf6"))))
(build-system font-build-system)
(home-page "http://tulrich.com/fonts/")
(synopsis "The Tuffy Truetype Font Family")
(description
"Thatcher Ulrich's first outline font design. He started with the goal of producing a neutral, readable sans-serif text font. There are lots of \"expressive\" fonts out there, but he wanted to start with something very plain and clean, something he might want to actually use. ")
(license license:public-domain)))
(define-public lief
(package
(name "python-lief")
(version "0.11.5")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/lief-project/LIEF.git")
(commit version)))
(file-name (git-file-name name version))
(sha256
(base32
"0qahjfg1n0x76ps2mbyljvws1l3qhkqvmxqbahps4qgywl2hbdkj"))))
(build-system python-build-system)
(native-inputs
`(("cmake" ,cmake)))
(home-page "https://github.com/lief-project/LIEF")
(synopsis "Library to Instrument Executable Formats")
(description "Python library to to provide a cross platform library which can
parse, modify and abstract ELF, PE and MachO formats.")
(license license:asl2.0)))
(define osslsigncode
(package
(name "osslsigncode")
(version "2.0")
(source (origin
(method url-fetch)
(uri (string-append "https://github.com/mtrojnar/"
name "/archive/" version ".tar.gz"))
(sha256
(base32
"0byri6xny770wwb2nciq44j5071122l14bvv65axdd70nfjf0q2s"))))
(build-system gnu-build-system)
(native-inputs
`(("pkg-config" ,pkg-config)
("autoconf" ,autoconf)
("automake" ,automake)
("libtool" ,libtool)))
(inputs
`(("openssl" ,openssl)))
(arguments
`(#:configure-flags
`("--without-gsf"
"--without-curl"
"--disable-dependency-tracking")))
(home-page "https://github.com/mtrojnar/osslsigncode")
(synopsis "Authenticode signing and timestamping tool")
(description "osslsigncode is a small tool that implements part of the
functionality of the Microsoft tool signtool.exe - more exactly the Authenticode
signing and timestamping. But osslsigncode is based on OpenSSL and cURL, and
thus should be able to compile on most platforms where these exist.")
(license license:gpl3+))) ; license is with openssl exception
(define-public python-asn1crypto
(package
(name "python-asn1crypto")
(version "1.4.0")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/wbond/asn1crypto")
(commit version)))
(file-name (git-file-name name version))
(sha256
(base32
"19abibn6jw20mzi1ln4n9jjvpdka8ygm4m439hplyrdfqbvgm01r"))))
(build-system python-build-system)
(arguments
'(#:phases
(modify-phases %standard-phases
(replace 'check
(lambda _
(invoke "python" "run.py" "tests"))))))
(home-page "https://github.com/wbond/asn1crypto")
(synopsis "ASN.1 parser and serializer in Python")
(description "asn1crypto is an ASN.1 parser and serializer with definitions
for private keys, public keys, certificates, CRL, OCSP, CMS, PKCS#3, PKCS#7,
PKCS#8, PKCS#12, PKCS#5, X.509 and TSP.")
(license license:expat)))
(define-public python-elfesteem
(let ((commit "87bbd79ab7e361004c98cc8601d4e5f029fd8bd5"))
(package
(name "python-elfesteem")
(version (git-version "0.1" "1" commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/LRGH/elfesteem")
(commit commit)))
(file-name (git-file-name name commit))
(sha256
(base32
"1nyvjisvyxyxnd0023xjf5846xd03lwawp5pfzr8vrky7wwm5maz"))))
(build-system python-build-system)
;; There are no tests, but attempting to run python setup.py test leads to
;; PYTHONPATH problems, just disable the test
(arguments '(#:tests? #f))
(home-page "https://github.com/LRGH/elfesteem")
(synopsis "ELF/PE/Mach-O parsing library")
(description "elfesteem parses ELF, PE and Mach-O files.")
(license license:lgpl2.1))))
(define-public python-oscrypto
(package
(name "python-oscrypto")
(version "1.2.1")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/wbond/oscrypto")
(commit version)))
(file-name (git-file-name name version))
(sha256
(base32
"1d4d8s4z340qhvb3g5m5v3436y3a71yc26wk4749q64m09kxqc3l"))
(patches (search-our-patches "oscrypto-hard-code-openssl.patch"))))
(build-system python-build-system)
(native-search-paths
(list (search-path-specification
(variable "SSL_CERT_FILE")
(file-type 'regular)
(separator #f) ;single entry
(files '("etc/ssl/certs/ca-certificates.crt")))))
(propagated-inputs
`(("python-asn1crypto" ,python-asn1crypto)
("openssl" ,openssl)))
(arguments
`(#:phases
(modify-phases %standard-phases
(add-after 'unpack 'hard-code-path-to-libscrypt
(lambda* (#:key inputs #:allow-other-keys)
(let ((openssl (assoc-ref inputs "openssl")))
(substitute* "oscrypto/__init__.py"
(("@GUIX_OSCRYPTO_USE_OPENSSL@")
(string-append openssl "/lib/libcrypto.so" "," openssl "/lib/libssl.so")))
#t)))
(add-after 'unpack 'disable-broken-tests
(lambda _
;; This test is broken as there is no keyboard interrupt.
(substitute* "tests/test_trust_list.py"
(("^(.*)class TrustListTests" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
(substitute* "tests/test_tls.py"
(("^(.*)class TLSTests" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
#t))
(replace 'check
(lambda _
(invoke "python" "run.py" "tests")
#t)))))
(home-page "https://github.com/wbond/oscrypto")
(synopsis "Compiler-free Python crypto library backed by the OS")
(description "oscrypto is a compilation-free, always up-to-date encryption library for Python.")
(license license:expat)))
(define-public python-oscryptotests
(package (inherit python-oscrypto)
(name "python-oscryptotests")
(arguments
`(#:tests? #f
#:phases
(modify-phases %standard-phases
(add-after 'unpack 'hard-code-path-to-libscrypt
(lambda* (#:key inputs #:allow-other-keys)
(chdir "tests")
#t)))))))
(define-public python-certvalidator
(let ((commit "e5bdb4bfcaa09fa0af355eb8867d00dfeecba08c"))
(package
(name "python-certvalidator")
(version (git-version "0.1" "1" commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/achow101/certvalidator")
(commit commit)))
(file-name (git-file-name name commit))
(sha256
(base32
"18pvxkvpkfkzgvfylv0kx65pmxfcv1hpsg03cip93krfvrrl4c75"))))
(build-system python-build-system)
(propagated-inputs
`(("python-asn1crypto" ,python-asn1crypto)
("python-oscrypto" ,python-oscrypto)
("python-oscryptotests", python-oscryptotests))) ;; certvalidator tests import oscryptotests
(arguments
`(#:phases
(modify-phases %standard-phases
(add-after 'unpack 'disable-broken-tests
(lambda _
(substitute* "tests/test_certificate_validator.py"
(("^(.*)class CertificateValidatorTests" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
(substitute* "tests/test_crl_client.py"
(("^(.*)def test_fetch_crl" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
(substitute* "tests/test_ocsp_client.py"
(("^(.*)def test_fetch_ocsp" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
(substitute* "tests/test_registry.py"
(("^(.*)def test_build_paths" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
(substitute* "tests/test_validate.py"
(("^(.*)def test_revocation_mode_hard" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
(substitute* "tests/test_validate.py"
(("^(.*)def test_revocation_mode_soft" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
#t))
(replace 'check
(lambda _
(invoke "python" "run.py" "tests")
#t)))))
(home-page "https://github.com/wbond/certvalidator")
(synopsis "Python library for validating X.509 certificates and paths")
(description "certvalidator is a Python library for validating X.509
certificates or paths. Supports various options, including: validation at a
specific moment in time, whitelisting and revocation checks.")
(license license:expat))))
(define-public python-requests-2.25.1
(package (inherit python-requests)
(version "2.25.1")
(source (origin
(method url-fetch)
(uri (pypi-uri "requests" version))
(sha256
(base32
"015qflyqsgsz09gnar69s6ga74ivq5kch69s4qxz3904m7a3v5r7"))))))
(define-public python-altgraph
(package
(name "python-altgraph")
(version "0.17")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/ronaldoussoren/altgraph")
(commit (string-append "v" version))))
(file-name (git-file-name name version))
(sha256
(base32
"09sm4srvvkw458pn48ga9q7ykr4xlz7q8gh1h9w7nxpf001qgpwb"))))
(build-system python-build-system)
(home-page "https://github.com/ronaldoussoren/altgraph")
(synopsis "Python graph (network) package")
(description "altgraph is a fork of graphlib: a graph (network) package for
constructing graphs, BFS and DFS traversals, topological sort, shortest paths,
etc. with graphviz output.")
(license license:expat)))
(define-public python-macholib
(package
(name "python-macholib")
(version "1.14")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/ronaldoussoren/macholib")
(commit (string-append "v" version))))
(file-name (git-file-name name version))
(sha256
(base32
"0aislnnfsza9wl4f0vp45ivzlc0pzhp9d4r08700slrypn5flg42"))))
(build-system python-build-system)
(propagated-inputs
`(("python-altgraph" ,python-altgraph)))
(arguments
'(#:phases
(modify-phases %standard-phases
(add-after 'unpack 'disable-broken-tests
(lambda _
;; This test is broken as there is no keyboard interrupt.
(substitute* "macholib_tests/test_command_line.py"
(("^(.*)class TestCmdLine" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line)))
(substitute* "macholib_tests/test_dyld.py"
(("^(.*)def test_\\S+_find" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line))
(("^(.*)def testBasic" line indent)
(string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line))
)
#t)))))
(home-page "https://github.com/ronaldoussoren/macholib")
(synopsis "Python library for analyzing and editing Mach-O headers")
(description "macholib is a Macho-O header analyzer and editor. It's
typically used as a dependency analysis tool, and also to rewrite dylib
references in Mach-O headers to be @executable_path relative. Though this tool
targets a platform specific file format, it is pure python code that is platform
and endian independent.")
(license license:expat)))
(define-public python-signapple
(let ((commit "b084cbbf44d5330448ffce0c7d118f75781b64bd"))
(package
(name "python-signapple")
(version (git-version "0.1" "1" commit))
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/achow101/signapple")
(commit commit)))
(file-name (git-file-name name commit))
(sha256
(base32
"0k7inccl2mzac3wq4asbr0kl8s4cghm8982z54kfascqg45shv01"))))
(build-system python-build-system)
(propagated-inputs
`(("python-asn1crypto" ,python-asn1crypto)
("python-oscrypto" ,python-oscrypto)
("python-certvalidator" ,python-certvalidator)
("python-elfesteem" ,python-elfesteem)
("python-requests" ,python-requests-2.25.1)
("python-macholib" ,python-macholib)
("libcrypto" ,openssl)))
;; There are no tests, but attempting to run python setup.py test leads to
;; problems, just disable the test
(arguments '(#:tests? #f))
(home-page "https://github.com/achow101/signapple")
(synopsis "Mach-O binary signature tool")
(description "signapple is a Python tool for creating, verifying, and
inspecting signatures in Mach-O binaries.")
(license license:expat))))
(define-public glibc-2.24
(package
(inherit glibc)
(version "2.24")
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://sourceware.org/git/glibc.git")
(commit "0d7f1ed30969886c8dde62fbf7d2c79967d4bace")))
(file-name (git-file-name "glibc" "0d7f1ed30969886c8dde62fbf7d2c79967d4bace"))
(sha256
(base32
"0g5hryia5v1k0qx97qffgwzrz4lr4jw3s5kj04yllhswsxyjbic3"))
(patches (search-our-patches "glibc-ldd-x86_64.patch"
"glibc-versioned-locpath.patch"
"glibc-2.24-elfm-loadaddr-dynamic-rewrite.patch"
"glibc-2.24-no-build-time-cxx-header-run.patch"))))))
(define glibc-2.27/digibyte-patched
(package-with-extra-patches glibc-2.27
(search-our-patches "glibc-2.27-riscv64-Use-__has_include__-to-include-asm-syscalls.h.patch")))
(packages->manifest
(append
(list ;; The Basics
bash
which
coreutils
util-linux
;; File(system) inspection
file
grep
diffutils
findutils
;; File transformation
patch
gawk
sed
moreutils
;; Compression and archiving
tar
bzip2
gzip
xz
;; Build tools
gnu-make
libtool
autoconf-2.71
automake
pkg-config
bison
;; Scripting
perl
python-3
;; Git
git
;; Tests
lief
;; Native gcc 7 toolchain
gcc-toolchain-7
(list gcc-toolchain-7 "static"))
(let ((target (getenv "HOST")))
(cond ((string-suffix? "-mingw32" target)
;; Windows
(list zip
(make-mingw-pthreads-cross-toolchain "x86_64-w64-mingw32")
(make-nsis-with-sde-support nsis-x86_64)
osslsigncode))
((string-contains target "-linux-")
(list (cond ((string-contains target "riscv64-")
(make-digibyte-cross-toolchain target
#:base-libc glibc-2.27/digibyte-patched
#:base-kernel-headers linux-libre-headers-4.19))
(else
(make-digibyte-cross-toolchain target)))))
((string-contains target "darwin")
(list clang-toolchain-10 binutils imagemagick libtiff librsvg font-tuffy cmake xorriso python-signapple))
(else '())))))

12
contrib/init/README.md Normal file
View file

@ -0,0 +1,12 @@
Sample configuration files for:
```
systemd: vextad.service
Upstart: vextad.conf
OpenRC: vextad.openrc
vextad.openrcconf
CentOS: vextad.init
macOS: org.vexta.vextad.plist
```
have been made available to assist packagers in creating node packages here.
See [doc/init.md](../../doc/init.md) for more information.

65
contrib/init/vextad.conf Normal file
View file

@ -0,0 +1,65 @@
description "Vexta Core Daemon"
start on runlevel [2345]
stop on starting rc RUNLEVEL=[016]
env VEXTAD_BIN="/usr/bin/vextad"
env VEXTAD_USER="vexta"
env VEXTAD_GROUP="vexta"
env VEXTAD_PIDDIR="/var/run/vextad"
# upstart can't handle variables constructed with other variables
env VEXTAD_PIDFILE="/var/run/vextad/vextad.pid"
env VEXTAD_CONFIGFILE="/etc/vexta/vexta.conf"
env VEXTAD_DATADIR="/var/lib/vextad"
expect fork
respawn
respawn limit 5 120
kill timeout 600
pre-start script
# this will catch non-existent config files
# vextad will check and exit with this very warning, but it can do so
# long after forking, leaving upstart to think everything started fine.
# since this is a commonly encountered case on install, just check and
# warn here.
if ! grep -qs '^rpcpassword=' "$VEXTAD_CONFIGFILE" ; then
echo "ERROR: You must set a secure rpcpassword to run vextad."
echo "The setting must appear in $VEXTAD_CONFIGFILE"
echo
echo "This password is security critical to securing wallets "
echo "and must not be the same as the rpcuser setting."
echo "You can generate a suitable random password using the following "
echo "command from the shell:"
echo
echo "bash -c 'tr -dc a-zA-Z0-9 < /dev/urandom | head -c32 && echo'"
echo
echo "It is recommended that you also set alertnotify so you are "
echo "notified of problems:"
echo
echo "ie: alertnotify=echo %%s | mail -s \"Vexta Alert\"" \
"admin@foo.com"
echo
exit 1
fi
mkdir -p "$VEXTAD_PIDDIR"
chmod 0755 "$VEXTAD_PIDDIR"
chown $VEXTAD_USER:$VEXTAD_GROUP "$VEXTAD_PIDDIR"
chown $VEXTAD_USER:$VEXTAD_GROUP "$VEXTAD_CONFIGFILE"
chmod 0660 "$VEXTAD_CONFIGFILE"
end script
exec start-stop-daemon \
--start \
--pidfile "$VEXTAD_PIDFILE" \
--chuid $VEXTAD_USER:$VEXTAD_GROUP \
--exec "$VEXTAD_BIN" \
-- \
-pid="$VEXTAD_PIDFILE" \
-conf="$VEXTAD_CONFIGFILE" \
-datadir="$VEXTAD_DATADIR" \
-disablewallet \
-daemon

67
contrib/init/vextad.init Normal file
View file

@ -0,0 +1,67 @@
#!/usr/bin/env bash
#
# vextad The vexta core server.
#
#
# chkconfig: 345 80 20
# description: vextad
# processname: vextad
#
# Source function library.
. /etc/init.d/functions
# you can override defaults in /etc/sysconfig/vextad, see below
if [ -f /etc/sysconfig/vextad ]; then
. /etc/sysconfig/vextad
fi
RETVAL=0
prog=vextad
# you can override the lockfile via VEXTAD_LOCKFILE in /etc/sysconfig/vextad
lockfile=${VEXTAD_LOCKFILE-/var/lock/subsys/vextad}
# vextad defaults to /usr/bin/vextad, override with VEXTAD_BIN
vextad=${VEXTAD_BIN-/usr/bin/vextad}
# vextad opts default to -disablewallet, override with VEXTAD_OPTS
vextad_opts=${VEXTAD_OPTS--disablewallet}
start() {
echo -n $"Starting $prog: "
daemon $DAEMONOPTS $vextad $vextad_opts
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch $lockfile
return $RETVAL
}
stop() {
echo -n $"Stopping $prog: "
killproc $prog -t600
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f $lockfile
return $RETVAL
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status $prog
;;
restart)
stop
start
;;
*)
echo "Usage: service $prog {start|stop|status|restart}"
exit 1
;;
esac

93
contrib/init/vextad.openrc Normal file
View file

@ -0,0 +1,93 @@
#!/sbin/openrc-run
# backward compatibility for existing gentoo layout
#
if [ -d "/var/lib/vexta/.vexta" ]; then
VEXTAD_DEFAULT_DATADIR="/var/lib/vexta/.vexta"
else
VEXTAD_DEFAULT_DATADIR="/var/lib/vextad"
fi
VEXTAD_CONFIGFILE=${VEXTAD_CONFIGFILE:-/etc/vexta/vexta.conf}
VEXTAD_PIDDIR=${VEXTAD_PIDDIR:-/var/run/vextad}
VEXTAD_PIDFILE=${VEXTAD_PIDFILE:-${VEXTAD_PIDDIR}/vextad.pid}
VEXTAD_DATADIR=${VEXTAD_DATADIR:-${VEXTAD_DEFAULT_DATADIR}}
VEXTAD_USER=${VEXTAD_USER:-${DIGIBYTE_USER:-vexta}}
VEXTAD_GROUP=${VEXTAD_GROUP:-vexta}
VEXTAD_BIN=${VEXTAD_BIN:-/usr/bin/vextad}
VEXTAD_NICE=${VEXTAD_NICE:-${NICELEVEL:-0}}
VEXTAD_OPTS="${VEXTAD_OPTS:-${DIGIBYTE_OPTS}}"
name="Vexta Core Daemon"
description="Vexta cryptocurrency P2P network daemon"
command="/usr/bin/vextad"
command_args="-pid=\"${VEXTAD_PIDFILE}\" \
-conf=\"${VEXTAD_CONFIGFILE}\" \
-datadir=\"${VEXTAD_DATADIR}\" \
-daemon \
${VEXTAD_OPTS}"
required_files="${VEXTAD_CONFIGFILE}"
start_stop_daemon_args="-u ${VEXTAD_USER} \
-N ${VEXTAD_NICE} -w 2000"
pidfile="${VEXTAD_PIDFILE}"
# The retry schedule to use when stopping the daemon. Could be either
# a timeout in seconds or multiple signal/timeout pairs (like
# "SIGKILL/180 SIGTERM/300")
retry="${VEXTAD_SIGTERM_TIMEOUT}"
depend() {
need localmount net
}
# verify
# 1) that the datadir exists and is writable (or create it)
# 2) that a directory for the pid exists and is writable
# 3) ownership and permissions on the config file
start_pre() {
checkpath \
-d \
--mode 0750 \
--owner "${VEXTAD_USER}:${VEXTAD_GROUP}" \
"${VEXTAD_DATADIR}"
checkpath \
-d \
--mode 0755 \
--owner "${VEXTAD_USER}:${VEXTAD_GROUP}" \
"${VEXTAD_PIDDIR}"
checkpath -f \
-o "${VEXTAD_USER}:${VEXTAD_GROUP}" \
-m 0660 \
"${VEXTAD_CONFIGFILE}"
checkconfig || return 1
}
checkconfig()
{
if grep -qs '^rpcuser=' "${VEXTAD_CONFIGFILE}" && \
! grep -qs '^rpcpassword=' "${VEXTAD_CONFIGFILE}" ; then
eerror ""
eerror "ERROR: You must set a secure rpcpassword to run vextad."
eerror "The setting must appear in ${VEXTAD_CONFIGFILE}"
eerror ""
eerror "This password is security critical to securing wallets "
eerror "and must not be the same as the rpcuser setting."
eerror "You can generate a suitable random password using the following "
eerror "command from the shell:"
eerror ""
eerror "bash -c 'tr -dc a-zA-Z0-9 < /dev/urandom | head -c32 && echo'"
eerror ""
eerror "It is recommended that you also set alertnotify so you are "
eerror "notified of problems:"
eerror ""
eerror "ie: alertnotify=echo %%s | mail -s \"Vexta Alert\"" \
"admin@foo.com"
eerror ""
return 1
fi
}

33
contrib/init/vextad.openrcconf Normal file
View file

@ -0,0 +1,33 @@
# /etc/conf.d/vextad: config file for /etc/init.d/vextad
# Config file location
#VEXTAD_CONFIGFILE="/etc/vexta/vexta.conf"
# What directory to write pidfile to? (created and owned by $VEXTAD_USER)
#VEXTAD_PIDDIR="/var/run/vextad"
# What filename to give the pidfile
#VEXTAD_PIDFILE="${VEXTAD_PIDDIR}/vextad.pid"
# Where to write vextad data (be mindful that the blockchain is large)
#VEXTAD_DATADIR="/var/lib/vextad"
# User and group to own vextad process
#VEXTAD_USER="vexta"
#VEXTAD_GROUP="vexta"
# Path to vextad executable
#VEXTAD_BIN="/usr/bin/vextad"
# Nice value to run vextad under
#VEXTAD_NICE=0
# Additional options (avoid -conf and -datadir, use flags above)
#VEXTAD_OPTS=""
# The timeout in seconds OpenRC will wait for vextad to terminate
# after a SIGTERM has been raised.
# Note that this will be mapped as argument to start-stop-daemon's
# '--retry' option, which means you can specify a retry schedule
# here. For more information see man 8 start-stop-daemon.
VEXTAD_SIGTERM_TIMEOUT=600

82
contrib/init/vextad.service Normal file
View file

@ -0,0 +1,82 @@
# It is not recommended to modify this file in-place, because it will
# be overwritten during package upgrades. If you want to add further
# options or overwrite existing ones then use
# $ systemctl edit vextad.service
# See "man systemd.service" for details.
# Note that almost all daemon options could be specified in
# /etc/vexta/vexta.conf, but keep in mind those explicitly
# specified as arguments in ExecStart= will override those in the
# config file.
[Unit]
Description=Vexta daemon
Documentation=https://github.com/vexta/vexta/blob/master/doc/init.md
# https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/bin/vextad -daemonwait \
-pid=/run/vextad/vextad.pid \
-conf=/etc/vexta/vexta.conf \
-datadir=/var/lib/vextad
# Make sure the config directory is readable by the service user
PermissionsStartOnly=true
ExecStartPre=/bin/chgrp vexta /etc/vexta
# Process management
####################
Type=forking
PIDFile=/run/vextad/vextad.pid
Restart=on-failure
TimeoutStartSec=infinity
TimeoutStopSec=600
# Directory creation and permissions
####################################
# Run as vexta:vexta
User=vexta
Group=vexta
# /run/vextad
RuntimeDirectory=vextad
RuntimeDirectoryMode=0710
# /etc/vexta
ConfigurationDirectory=vexta
ConfigurationDirectoryMode=0710
# /var/lib/vextad
StateDirectory=vextad
StateDirectoryMode=0710
# Hardening measures
####################
# Provide a private /tmp and /var/tmp.
PrivateTmp=true
# Mount /usr, /boot/ and /etc read-only for the process.
ProtectSystem=full
# Deny access to /home, /root and /run/user
ProtectHome=true
# Disallow the process and all of its children to gain
# new privileges through execve().
NoNewPrivileges=true
# Use a new /dev namespace only populated with API pseudo devices
# such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true
# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true
[Install]
WantedBy=multi-user.target

25
contrib/init/vextad.service.app.service Normal file
View file

@ -0,0 +1,25 @@
# It is not recommended to modify this file in-place, because it will
# be overwritten during package upgrades. If you want to add further
# options or overwrite existing ones then use
# $ systemctl edit vextad.service
# See "man systemd.service" for details.
# Note that almost all daemon options could be specified in
# /etc/vexta/vexta.conf
[Unit]
Description=Vexta daemon
After=network.target
[Service]
ExecStart=/usr/bin/vextad -daemon -conf=/etc/vexta/vexta.conf -pid=/run/vextad/vextad.pid
# Creates /run/vextad owned by vexta
RuntimeDirectory=vextad
User=vexta
Type=forking
PIDFile=/run/vextad/vextad.pid
Restart=on-failure
PrivateTmp=true
[Install]
WantedBy=multi-user.target

251
contrib/install_db4.sh Executable file
View file

@ -0,0 +1,251 @@
#!/bin/sh
# Copyright (c) 2017-2019 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
# Install libdb4.8 (Berkeley DB).
export LC_ALL=C
set -e
if [ -z "${1}" ]; then
echo "Usage: $0 <base-dir> [<extra-bdb-configure-flag> ...]"
echo
echo "Must specify a single argument: the directory in which db4 will be built."
echo "This is probably \`pwd\` if you're at the root of the digibyte repository."
exit 1
fi
expand_path() {
cd "${1}" && pwd -P
}
BDB_PREFIX="$(expand_path ${1})/db4"; shift;
BDB_VERSION='db-4.8.30.NC'
BDB_HASH='12edc0df75bf9abd7f82f821795bcee50f42cb2e5f76a6a281b85732798364ef'
BDB_URL="https://download.oracle.com/berkeley-db/${BDB_VERSION}.tar.gz"
check_exists() {
command -v "$1" >/dev/null
}
sha256_check() {
# Args: <sha256_hash> <filename>
#
if check_exists sha256sum; then
echo "${1} ${2}" | sha256sum -c
elif check_exists sha256; then
if [ "$(uname)" = "FreeBSD" ]; then
sha256 -c "${1}" "${2}"
else
echo "${1} ${2}" | sha256 -c
fi
else
echo "${1} ${2}" | shasum -a 256 -c
fi
}
http_get() {
# Args: <url> <filename> <sha256_hash>
#
# It's acceptable that we don't require SSL here because we manually verify
# content hashes below.
#
if [ -f "${2}" ]; then
echo "File ${2} already exists; not downloading again"
elif check_exists curl; then
curl --insecure --retry 5 "${1}" -o "${2}"
else
wget --no-check-certificate "${1}" -O "${2}"
fi
sha256_check "${3}" "${2}"
}
mkdir -p "${BDB_PREFIX}"
http_get "${BDB_URL}" "${BDB_VERSION}.tar.gz" "${BDB_HASH}"
tar -xzvf ${BDB_VERSION}.tar.gz -C "$BDB_PREFIX"
cd "${BDB_PREFIX}/${BDB_VERSION}/"
# Apply a patch necessary when building with clang and c++11 (see https://community.oracle.com/thread/3952592)
patch --ignore-whitespace -p1 << 'EOF'
commit 3311d68f11d1697565401eee6efc85c34f022ea7
Author: fanquake <fanquake@gmail.com>
Date: Mon Aug 17 20:03:56 2020 +0800
Fix C++11 compatibility
diff --git a/dbinc/atomic.h b/dbinc/atomic.h
index 0034dcc..7c11d4a 100644
--- a/dbinc/atomic.h
+++ b/dbinc/atomic.h
@@ -70,7 +70,7 @@ typedef struct {
* These have no memory barriers; the caller must include them when necessary.
*/
#define atomic_read(p) ((p)->value)
-#define atomic_init(p, val) ((p)->value = (val))
+#define atomic_init_db(p, val) ((p)->value = (val))
#ifdef HAVE_ATOMIC_SUPPORT
@@ -144,7 +144,7 @@ typedef LONG volatile *interlocked_val;
#define atomic_inc(env, p) __atomic_inc(p)
#define atomic_dec(env, p) __atomic_dec(p)
#define atomic_compare_exchange(env, p, o, n) \
- __atomic_compare_exchange((p), (o), (n))
+ __atomic_compare_exchange_db((p), (o), (n))
static inline int __atomic_inc(db_atomic_t *p)
{
int temp;
@@ -176,7 +176,7 @@ static inline int __atomic_dec(db_atomic_t *p)
* http://gcc.gnu.org/onlinedocs/gcc-4.1.0/gcc/Atomic-Builtins.html
* which configure could be changed to use.
*/
-static inline int __atomic_compare_exchange(
+static inline int __atomic_compare_exchange_db(
db_atomic_t *p, atomic_value_t oldval, atomic_value_t newval)
{
atomic_value_t was;
@@ -206,7 +206,7 @@ static inline int __atomic_compare_exchange(
#define atomic_dec(env, p) (--(p)->value)
#define atomic_compare_exchange(env, p, oldval, newval) \
(DB_ASSERT(env, atomic_read(p) == (oldval)), \
- atomic_init(p, (newval)), 1)
+ atomic_init_db(p, (newval)), 1)
#else
#define atomic_inc(env, p) __atomic_inc(env, p)
#define atomic_dec(env, p) __atomic_dec(env, p)
diff --git a/mp/mp_fget.c b/mp/mp_fget.c
index 5fdee5a..0b75f57 100644
--- a/mp/mp_fget.c
+++ b/mp/mp_fget.c
@@ -617,7 +617,7 @@ alloc: /* Allocate a new buffer header and data space. */
/* Initialize enough so we can call __memp_bhfree. */
alloc_bhp->flags = 0;
- atomic_init(&alloc_bhp->ref, 1);
+ atomic_init_db(&alloc_bhp->ref, 1);
#ifdef DIAGNOSTIC
if ((uintptr_t)alloc_bhp->buf & (sizeof(size_t) - 1)) {
__db_errx(env,
@@ -911,7 +911,7 @@ alloc: /* Allocate a new buffer header and data space. */
MVCC_MPROTECT(bhp->buf, mfp->stat.st_pagesize,
PROT_READ);
- atomic_init(&alloc_bhp->ref, 1);
+ atomic_init_db(&alloc_bhp->ref, 1);
MUTEX_LOCK(env, alloc_bhp->mtx_buf);
alloc_bhp->priority = bhp->priority;
alloc_bhp->pgno = bhp->pgno;
diff --git a/mp/mp_mvcc.c b/mp/mp_mvcc.c
index 34467d2..f05aa0c 100644
--- a/mp/mp_mvcc.c
+++ b/mp/mp_mvcc.c
@@ -276,7 +276,7 @@ __memp_bh_freeze(dbmp, infop, hp, bhp, need_frozenp)
#else
memcpy(frozen_bhp, bhp, SSZA(BH, buf));
#endif
- atomic_init(&frozen_bhp->ref, 0);
+ atomic_init_db(&frozen_bhp->ref, 0);
if (mutex != MUTEX_INVALID)
frozen_bhp->mtx_buf = mutex;
else if ((ret = __mutex_alloc(env, MTX_MPOOL_BH,
@@ -428,7 +428,7 @@ __memp_bh_thaw(dbmp, infop, hp, frozen_bhp, alloc_bhp)
#endif
alloc_bhp->mtx_buf = mutex;
MUTEX_LOCK(env, alloc_bhp->mtx_buf);
- atomic_init(&alloc_bhp->ref, 1);
+ atomic_init_db(&alloc_bhp->ref, 1);
F_CLR(alloc_bhp, BH_FROZEN);
}
diff --git a/mp/mp_region.c b/mp/mp_region.c
index e6cece9..ddbe906 100644
--- a/mp/mp_region.c
+++ b/mp/mp_region.c
@@ -224,7 +224,7 @@ __memp_init(env, dbmp, reginfo_off, htab_buckets, max_nreg)
MTX_MPOOL_FILE_BUCKET, 0, &htab[i].mtx_hash)) != 0)
return (ret);
SH_TAILQ_INIT(&htab[i].hash_bucket);
- atomic_init(&htab[i].hash_page_dirty, 0);
+ atomic_init_db(&htab[i].hash_page_dirty, 0);
}
/*
@@ -269,7 +269,7 @@ __memp_init(env, dbmp, reginfo_off, htab_buckets, max_nreg)
hp->mtx_hash = (mtx_base == MUTEX_INVALID) ? MUTEX_INVALID :
mtx_base + i;
SH_TAILQ_INIT(&hp->hash_bucket);
- atomic_init(&hp->hash_page_dirty, 0);
+ atomic_init_db(&hp->hash_page_dirty, 0);
#ifdef HAVE_STATISTICS
hp->hash_io_wait = 0;
hp->hash_frozen = hp->hash_thawed = hp->hash_frozen_freed = 0;
diff --git a/mutex/mut_method.c b/mutex/mut_method.c
index 2588763..5c6d516 100644
--- a/mutex/mut_method.c
+++ b/mutex/mut_method.c
@@ -426,7 +426,7 @@ atomic_compare_exchange(env, v, oldval, newval)
MUTEX_LOCK(env, mtx);
ret = atomic_read(v) == oldval;
if (ret)
- atomic_init(v, newval);
+ atomic_init_db(v, newval);
MUTEX_UNLOCK(env, mtx);
return (ret);
diff --git a/mutex/mut_tas.c b/mutex/mut_tas.c
index f3922e0..e40fcdf 100644
--- a/mutex/mut_tas.c
+++ b/mutex/mut_tas.c
@@ -46,7 +46,7 @@ __db_tas_mutex_init(env, mutex, flags)
#ifdef HAVE_SHARED_LATCHES
if (F_ISSET(mutexp, DB_MUTEX_SHARED))
- atomic_init(&mutexp->sharecount, 0);
+ atomic_init_db(&mutexp->sharecount, 0);
else
#endif
if (MUTEX_INIT(&mutexp->tas)) {
@@ -486,7 +486,7 @@ __db_tas_mutex_unlock(env, mutex)
F_CLR(mutexp, DB_MUTEX_LOCKED);
/* Flush flag update before zeroing count */
MEMBAR_EXIT();
- atomic_init(&mutexp->sharecount, 0);
+ atomic_init_db(&mutexp->sharecount, 0);
} else {
DB_ASSERT(env, sharecount > 0);
MEMBAR_EXIT();
EOF
# The packaged config.guess and config.sub are ancient (2009) and can cause build issues.
# Replace them with modern versions.
# See https://github.com/digibyte/digibyte/issues/16064
CONFIG_GUESS_URL='https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=4550d2f15b3a7ce2451c1f29500b9339430c877f'
CONFIG_GUESS_HASH='c8f530e01840719871748a8071113435bdfdf75b74c57e78e47898edea8754ae'
CONFIG_SUB_URL='https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=4550d2f15b3a7ce2451c1f29500b9339430c877f'
CONFIG_SUB_HASH='3969f7d5f6967ccc6f792401b8ef3916a1d1b1d0f0de5a4e354c95addb8b800e'
rm -f "dist/config.guess"
rm -f "dist/config.sub"
http_get "${CONFIG_GUESS_URL}" dist/config.guess "${CONFIG_GUESS_HASH}"
http_get "${CONFIG_SUB_URL}" dist/config.sub "${CONFIG_SUB_HASH}"
cd build_unix/
"${BDB_PREFIX}/${BDB_VERSION}/dist/configure" \
--enable-cxx --disable-shared --disable-replication --with-pic --prefix="${BDB_PREFIX}" \
"${@}"
make install
echo
echo "db4 build complete."
echo
# shellcheck disable=SC2016
echo 'When compiling digibyted, run `./configure` in the following way:'
echo
echo " export BDB_PREFIX='${BDB_PREFIX}'"
# shellcheck disable=SC2016
echo ' ./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include" ...'

54
contrib/linearize/README.md Normal file
View file

@ -0,0 +1,54 @@
# Linearize
Construct a linear, no-fork, best version of the DigiByte blockchain.
## Step 1: Download hash list
$ ./linearize-hashes.py linearize.cfg > hashlist.txt
Required configuration file settings for linearize-hashes:
* RPC: `datadir` (Required if `rpcuser` and `rpcpassword` are not specified)
* RPC: `rpcuser`, `rpcpassword` (Required if `datadir` is not specified)
Optional config file setting for linearize-hashes:
* RPC: `host` (Default: `127.0.0.1`)
* RPC: `port` (Default: `8332`)
* Blockchain: `min_height`, `max_height`
* `rev_hash_bytes`: If true, the written block hash list will be
byte-reversed. (In other words, the hash returned by getblockhash will have its
bytes reversed.) False by default. Intended for generation of
standalone hash lists but safe to use with linearize-data.py, which will output
the same data no matter which byte format is chosen.
The `linearize-hashes` script requires a connection, local or remote, to a
JSON-RPC server. Running `digibyted` or `digibyte-qt -server` will be sufficient.
## Step 2: Copy local block data
$ ./linearize-data.py linearize.cfg
Required configuration file settings:
* `output_file`: The file that will contain the final blockchain.
or
* `output`: Output directory for linearized `blocks/blkNNNNN.dat` output.
Optional config file setting for linearize-data:
* `debug_output`: Some printouts may not always be desired. If true, such output
will be printed.
* `file_timestamp`: Set each file's last-accessed and last-modified times,
respectively, to the current time and to the timestamp of the most recent block
written to the script's blockchain.
* `genesis`: The hash of the genesis block in the blockchain.
* `input`: digibyted blocks/ directory containing blkNNNNN.dat
* `hashlist`: text file containing list of block hashes created by
linearize-hashes.py.
* `max_out_sz`: Maximum size for files created by the `output_file` option.
(Default: `1000*1000*1000 bytes`)
* `netmagic`: Network magic number.
* `out_of_order_cache_sz`: If out-of-order blocks are being read, the block can
be written to a cache so that the blockchain doesn't have to be sought again.
This option specifies the cache size. (Default: `100*1000*1000 bytes`)
* `rev_hash_bytes`: If true, the block hash list written by linearize-hashes.py
will be byte-reversed when read by linearize-data.py. See the linearize-hashes
entry for more information.
* `split_timestamp`: Split blockchain files when a new month is first seen, in
addition to reaching a maximum file size (`max_out_sz`).

63
contrib/linearize/example-linearize.cfg Normal file
View file

@ -0,0 +1,63 @@
# digibyted RPC settings (linearize-hashes)
rpcuser=someuser
rpcpassword=somepassword
#datadir=~/.digibyte
host=127.0.0.1
#mainnet default
port=8332
#testnet default
#port=18332
#regtest default
#port=18443
#signet default
#port=38332
# bootstrap.dat hashlist settings (linearize-hashes)
max_height=313000
# bootstrap.dat input/output settings (linearize-data)
# mainnet
netmagic=f9beb4d9
genesis=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
input=/home/example/.digibyte/blocks
# testnet
#netmagic=0b110907
#genesis=000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943
#input=/home/example/.digibyte/testnet3/blocks
# regtest
#netmagic=fabfb5da
#genesis=0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
#input=/home/example/.digibyte/regtest/blocks
# signet
#netmagic=0a03cf40
#genesis=00000008819873e925422c1ff0f99f7cc9bbb232af63a077a480a3633bee1ef6
#input=/home/example/.digibyte/signet/blocks
# "output" option causes blockchain files to be written to the given location,
# with "output_file" ignored. If not used, "output_file" is used instead.
# output=/home/example/blockchain_directory
output_file=/home/example/Downloads/bootstrap.dat
hashlist=hashlist.txt
# Maximum size in bytes of out-of-order blocks cache in memory
out_of_order_cache_sz = 100000000
# Do we want the reverse the hash bytes coming from getblockhash?
rev_hash_bytes = False
# On a new month, do we want to set the access and modify times of the new
# blockchain file?
file_timestamp = 0
# Do we want to split the blockchain files given a new month or specific height?
split_timestamp = 0
# Do we want debug printouts?
debug_output = False

351
contrib/linearize/linearize-data.py Executable file
View file

@ -0,0 +1,351 @@
#!/usr/bin/env python3
#
# linearize-data.py: Construct a linear, no-fork version of the chain.
#
# Copyright (c) 2013-2018 The Bitcoin Core developers
# Copyright (c) 2013-2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
import struct
import re
import os
import os.path
import sys
import hashlib
import datetime
import time
import glob
from collections import namedtuple
from binascii import unhexlify
settings = {}
def hex_switchEndian(s):
""" Switches the endianness of a hex string (in pairs of hex chars) """
pairList = [s[i:i+2].encode() for i in range(0, len(s), 2)]
return b''.join(pairList[::-1]).decode()
def uint32(x):
return x & 0xffffffff
def bytereverse(x):
return uint32(( ((x) << 24) | (((x) << 8) & 0x00ff0000) |
(((x) >> 8) & 0x0000ff00) | ((x) >> 24) ))
def bufreverse(in_buf):
out_words = []
for i in range(0, len(in_buf), 4):
word = struct.unpack('@I', in_buf[i:i+4])[0]
out_words.append(struct.pack('@I', bytereverse(word)))
return b''.join(out_words)
def wordreverse(in_buf):
out_words = []
for i in range(0, len(in_buf), 4):
out_words.append(in_buf[i:i+4])
out_words.reverse()
return b''.join(out_words)
def calc_hdr_hash(blk_hdr):
hash1 = hashlib.sha256()
hash1.update(blk_hdr)
hash1_o = hash1.digest()
hash2 = hashlib.sha256()
hash2.update(hash1_o)
hash2_o = hash2.digest()
return hash2_o
def calc_hash_str(blk_hdr):
hash = calc_hdr_hash(blk_hdr)
hash = bufreverse(hash)
hash = wordreverse(hash)
hash_str = hash.hex()
return hash_str
def get_blk_dt(blk_hdr):
members = struct.unpack("<I", blk_hdr[68:68+4])
nTime = members[0]
dt = datetime.datetime.fromtimestamp(nTime)
dt_ym = datetime.datetime(dt.year, dt.month, 1)
return (dt_ym, nTime)
# When getting the list of block hashes, undo any byte reversals.
def get_block_hashes(settings):
blkindex = []
f = open(settings['hashlist'], "r", encoding="utf8")
for line in f:
line = line.rstrip()
if settings['rev_hash_bytes'] == 'true':
line = hex_switchEndian(line)
blkindex.append(line)
print("Read " + str(len(blkindex)) + " hashes")
return blkindex
# The block map shouldn't give or receive byte-reversed hashes.
def mkblockmap(blkindex):
blkmap = {}
for height,hash in enumerate(blkindex):
blkmap[hash] = height
return blkmap
# This gets the first block file ID that exists from the input block
# file directory.
def getFirstBlockFileId(block_dir_path):
# First, this sets up a pattern to search for block files, for
# example 'blkNNNNN.dat'.
blkFilePattern = os.path.join(block_dir_path, "blk[0-9][0-9][0-9][0-9][0-9].dat")
# This search is done with glob
blkFnList = glob.glob(blkFilePattern)
if len(blkFnList) == 0:
print("blocks not pruned - starting at 0")
return 0
# We then get the lexicographic minimum, which should be the first
# block file name.
firstBlkFilePath = min(blkFnList)
firstBlkFn = os.path.basename(firstBlkFilePath)
# now, the string should be ['b','l','k','N','N','N','N','N','.','d','a','t']
# So get the ID by choosing: 3 4 5 6 7
# The ID is not necessarily 0 if this is a pruned node.
blkId = int(firstBlkFn[3:8])
return blkId
# Block header and extent on disk
BlockExtent = namedtuple('BlockExtent', ['fn', 'offset', 'inhdr', 'blkhdr', 'size'])
class BlockDataCopier:
def __init__(self, settings, blkindex, blkmap):
self.settings = settings
self.blkindex = blkindex
self.blkmap = blkmap
# Get first occurring block file id - for pruned nodes this
# will not necessarily be 0
self.inFn = getFirstBlockFileId(self.settings['input'])
self.inF = None
self.outFn = 0
self.outsz = 0
self.outF = None
self.outFname = None
self.blkCountIn = 0
self.blkCountOut = 0
self.lastDate = datetime.datetime(2000, 1, 1)
self.highTS = 1408893517 - 315360000
self.timestampSplit = False
self.fileOutput = True
self.setFileTime = False
self.maxOutSz = settings['max_out_sz']
if 'output' in settings:
self.fileOutput = False
if settings['file_timestamp'] != 0:
self.setFileTime = True
if settings['split_timestamp'] != 0:
self.timestampSplit = True
# Extents and cache for out-of-order blocks
self.blockExtents = {}
self.outOfOrderData = {}
self.outOfOrderSize = 0 # running total size for items in outOfOrderData
def writeBlock(self, inhdr, blk_hdr, rawblock):
blockSizeOnDisk = len(inhdr) + len(blk_hdr) + len(rawblock)
if not self.fileOutput and ((self.outsz + blockSizeOnDisk) > self.maxOutSz):
self.outF.close()
if self.setFileTime:
os.utime(self.outFname, (int(time.time()), self.highTS))
self.outF = None
self.outFname = None
self.outFn = self.outFn + 1
self.outsz = 0
(blkDate, blkTS) = get_blk_dt(blk_hdr)
if self.timestampSplit and (blkDate > self.lastDate):
print("New month " + blkDate.strftime("%Y-%m") + " @ " + self.hash_str)
self.lastDate = blkDate
if self.outF:
self.outF.close()
if self.setFileTime:
os.utime(self.outFname, (int(time.time()), self.highTS))
self.outF = None
self.outFname = None
self.outFn = self.outFn + 1
self.outsz = 0
if not self.outF:
if self.fileOutput:
self.outFname = self.settings['output_file']
else:
self.outFname = os.path.join(self.settings['output'], "blk%05d.dat" % self.outFn)
print("Output file " + self.outFname)
self.outF = open(self.outFname, "wb")
self.outF.write(inhdr)
self.outF.write(blk_hdr)
self.outF.write(rawblock)
self.outsz = self.outsz + len(inhdr) + len(blk_hdr) + len(rawblock)
self.blkCountOut = self.blkCountOut + 1
if blkTS > self.highTS:
self.highTS = blkTS
if (self.blkCountOut % 1000) == 0:
print('%i blocks scanned, %i blocks written (of %i, %.1f%% complete)' %
(self.blkCountIn, self.blkCountOut, len(self.blkindex), 100.0 * self.blkCountOut / len(self.blkindex)))
def inFileName(self, fn):
return os.path.join(self.settings['input'], "blk%05d.dat" % fn)
def fetchBlock(self, extent):
'''Fetch block contents from disk given extents'''
with open(self.inFileName(extent.fn), "rb") as f:
f.seek(extent.offset)
return f.read(extent.size)
def copyOneBlock(self):
'''Find the next block to be written in the input, and copy it to the output.'''
extent = self.blockExtents.pop(self.blkCountOut)
if self.blkCountOut in self.outOfOrderData:
# If the data is cached, use it from memory and remove from the cache
rawblock = self.outOfOrderData.pop(self.blkCountOut)
self.outOfOrderSize -= len(rawblock)
else: # Otherwise look up data on disk
rawblock = self.fetchBlock(extent)
self.writeBlock(extent.inhdr, extent.blkhdr, rawblock)
def run(self):
while self.blkCountOut < len(self.blkindex):
if not self.inF:
fname = self.inFileName(self.inFn)
print("Input file " + fname)
try:
self.inF = open(fname, "rb")
except IOError:
print("Premature end of block data")
return
inhdr = self.inF.read(8)
if (not inhdr or (inhdr[0] == "\0")):
self.inF.close()
self.inF = None
self.inFn = self.inFn + 1
continue
inMagic = inhdr[:4]
if (inMagic != self.settings['netmagic']):
# Seek backwards 7 bytes (skipping the first byte in the previous search)
# and continue searching from the new position if the magic bytes are not
# found.
self.inF.seek(-7, os.SEEK_CUR)
continue
inLenLE = inhdr[4:]
su = struct.unpack("<I", inLenLE)
inLen = su[0] - 80 # length without header
blk_hdr = self.inF.read(80)
inExtent = BlockExtent(self.inFn, self.inF.tell(), inhdr, blk_hdr, inLen)
self.hash_str = calc_hash_str(blk_hdr)
if not self.hash_str in blkmap:
# Because blocks can be written to files out-of-order as of 0.10, the script
# may encounter blocks it doesn't know about. Treat as debug output.
if settings['debug_output'] == 'true':
print("Skipping unknown block " + self.hash_str)
self.inF.seek(inLen, os.SEEK_CUR)
continue
blkHeight = self.blkmap[self.hash_str]
self.blkCountIn += 1
if self.blkCountOut == blkHeight:
# If in-order block, just copy
rawblock = self.inF.read(inLen)
self.writeBlock(inhdr, blk_hdr, rawblock)
# See if we can catch up to prior out-of-order blocks
while self.blkCountOut in self.blockExtents:
self.copyOneBlock()
else: # If out-of-order, skip over block data for now
self.blockExtents[blkHeight] = inExtent
if self.outOfOrderSize < self.settings['out_of_order_cache_sz']:
# If there is space in the cache, read the data
# Reading the data in file sequence instead of seeking and fetching it later is preferred,
# but we don't want to fill up memory
self.outOfOrderData[blkHeight] = self.inF.read(inLen)
self.outOfOrderSize += inLen
else: # If no space in cache, seek forward
self.inF.seek(inLen, os.SEEK_CUR)
print("Done (%i blocks written)" % (self.blkCountOut))
if __name__ == '__main__':
if len(sys.argv) != 2:
print("Usage: linearize-data.py CONFIG-FILE")
sys.exit(1)
f = open(sys.argv[1], encoding="utf8")
for line in f:
# skip comment lines
m = re.search(r'^\s*#', line)
if m:
continue
# parse key=value lines
m = re.search(r'^(\w+)\s*=\s*(\S.*)$', line)
if m is None:
continue
settings[m.group(1)] = m.group(2)
f.close()
# Force hash byte format setting to be lowercase to make comparisons easier.
# Also place upfront in case any settings need to know about it.
if 'rev_hash_bytes' not in settings:
settings['rev_hash_bytes'] = 'false'
settings['rev_hash_bytes'] = settings['rev_hash_bytes'].lower()
if 'netmagic' not in settings:
settings['netmagic'] = 'f9beb4d9'
if 'genesis' not in settings:
settings['genesis'] = '000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f'
if 'input' not in settings:
settings['input'] = 'input'
if 'hashlist' not in settings:
settings['hashlist'] = 'hashlist.txt'
if 'file_timestamp' not in settings:
settings['file_timestamp'] = 0
if 'split_timestamp' not in settings:
settings['split_timestamp'] = 0
if 'max_out_sz' not in settings:
settings['max_out_sz'] = 1000 * 1000 * 1000
if 'out_of_order_cache_sz' not in settings:
settings['out_of_order_cache_sz'] = 100 * 1000 * 1000
if 'debug_output' not in settings:
settings['debug_output'] = 'false'
settings['max_out_sz'] = int(settings['max_out_sz'])
settings['split_timestamp'] = int(settings['split_timestamp'])
settings['file_timestamp'] = int(settings['file_timestamp'])
settings['netmagic'] = unhexlify(settings['netmagic'].encode('utf-8'))
settings['out_of_order_cache_sz'] = int(settings['out_of_order_cache_sz'])
settings['debug_output'] = settings['debug_output'].lower()
if 'output_file' not in settings and 'output' not in settings:
print("Missing output file / directory")
sys.exit(1)
blkindex = get_block_hashes(settings)
blkmap = mkblockmap(blkindex)
# Block hash map won't be byte-reversed. Neither should the genesis hash.
if not settings['genesis'] in blkmap:
print("Genesis block not found in hashlist")
else:
BlockDataCopier(settings, blkindex, blkmap).run()

153
contrib/linearize/linearize-hashes.py Executable file
View file

@ -0,0 +1,153 @@
#!/usr/bin/env python3
#
# linearize-hashes.py: List blocks in a linear, no-fork version of the chain.
#
# Copyright (c) 2013-2018 The Bitcoin Core developers
# Copyright (c) 2013-2019 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
from http.client import HTTPConnection
import json
import re
import base64
import sys
import os
import os.path
settings = {}
def hex_switchEndian(s):
""" Switches the endianness of a hex string (in pairs of hex chars) """
pairList = [s[i:i+2].encode() for i in range(0, len(s), 2)]
return b''.join(pairList[::-1]).decode()
class DigiByteRPC:
def __init__(self, host, port, username, password):
authpair = "%s:%s" % (username, password)
authpair = authpair.encode('utf-8')
self.authhdr = b"Basic " + base64.b64encode(authpair)
self.conn = HTTPConnection(host, port=port, timeout=30)
def execute(self, obj):
try:
self.conn.request('POST', '/', json.dumps(obj),
{ 'Authorization' : self.authhdr,
'Content-type' : 'application/json' })
except ConnectionRefusedError:
print('RPC connection refused. Check RPC settings and the server status.',
file=sys.stderr)
return None
resp = self.conn.getresponse()
if resp is None:
print("JSON-RPC: no response", file=sys.stderr)
return None
body = resp.read().decode('utf-8')
resp_obj = json.loads(body)
return resp_obj
@staticmethod
def build_request(idx, method, params):
obj = { 'version' : '1.1',
'method' : method,
'id' : idx }
if params is None:
obj['params'] = []
else:
obj['params'] = params
return obj
@staticmethod
def response_is_error(resp_obj):
return 'error' in resp_obj and resp_obj['error'] is not None
def get_block_hashes(settings, max_blocks_per_call=10000):
rpc = DigiByteRPC(settings['host'], settings['port'],
settings['rpcuser'], settings['rpcpassword'])
height = settings['min_height']
while height < settings['max_height']+1:
num_blocks = min(settings['max_height']+1-height, max_blocks_per_call)
batch = []
for x in range(num_blocks):
batch.append(rpc.build_request(x, 'getblockhash', [height + x]))
reply = rpc.execute(batch)
if reply is None:
print('Cannot continue. Program will halt.')
return None
for x,resp_obj in enumerate(reply):
if rpc.response_is_error(resp_obj):
print('JSON-RPC: error at height', height+x, ': ', resp_obj['error'], file=sys.stderr)
sys.exit(1)
assert(resp_obj['id'] == x) # assume replies are in-sequence
if settings['rev_hash_bytes'] == 'true':
resp_obj['result'] = hex_switchEndian(resp_obj['result'])
print(resp_obj['result'])
height += num_blocks
def get_rpc_cookie():
# Open the cookie file
with open(os.path.join(os.path.expanduser(settings['datadir']), '.cookie'), 'r', encoding="ascii") as f:
combined = f.readline()
combined_split = combined.split(":")
settings['rpcuser'] = combined_split[0]
settings['rpcpassword'] = combined_split[1]
if __name__ == '__main__':
if len(sys.argv) != 2:
print("Usage: linearize-hashes.py CONFIG-FILE")
sys.exit(1)
f = open(sys.argv[1], encoding="utf8")
for line in f:
# skip comment lines
m = re.search(r'^\s*#', line)
if m:
continue
# parse key=value lines
m = re.search(r'^(\w+)\s*=\s*(\S.*)$', line)
if m is None:
continue
settings[m.group(1)] = m.group(2)
f.close()
if 'host' not in settings:
settings['host'] = '127.0.0.1'
if 'port' not in settings:
settings['port'] = 8332
if 'min_height' not in settings:
settings['min_height'] = 0
if 'max_height' not in settings:
settings['max_height'] = 313000
if 'rev_hash_bytes' not in settings:
settings['rev_hash_bytes'] = 'false'
use_userpass = True
use_datadir = False
if 'rpcuser' not in settings or 'rpcpassword' not in settings:
use_userpass = False
if 'datadir' in settings and not use_userpass:
use_datadir = True
if not use_userpass and not use_datadir:
print("Missing datadir or username and/or password in cfg file", file=sys.stderr)
sys.exit(1)
settings['port'] = int(settings['port'])
settings['min_height'] = int(settings['min_height'])
settings['max_height'] = int(settings['max_height'])
# Force hash byte format setting to be lowercase to make comparisons easier.
settings['rev_hash_bytes'] = settings['rev_hash_bytes'].lower()
# Get the rpc user and pass from the cookie if the datadir is set
if use_datadir:
get_rpc_cookie()
get_block_hashes(settings)

674
contrib/macdeploy/LICENSE Normal file
View file

@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.

122
contrib/macdeploy/README.md Normal file
View file

@ -0,0 +1,122 @@
# MacOS Deployment
The `macdeployqtplus` script should not be run manually. Instead, after building as usual:
```bash
make deploy
```
When complete, it will have produced `DigiByte-Core.dmg`.
## SDK Extraction
### Step 1: Obtaining `Xcode.app`
Our current macOS SDK
(`Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz`) can be
extracted from
[Xcode_12.2.xip](https://download.developer.apple.com/Developer_Tools/Xcode_12.2/Xcode_12.2.xip).
Alternatively, after logging in to your account go to 'Downloads', then 'More'
and look for [`Xcode_12.2`](https://download.developer.apple.com/Developer_Tools/Xcode_12.2/Xcode_12.2.xip).
An Apple ID and cookies enabled for the hostname are needed to download this.
The `sha256sum` of the archive should be `28d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0`.
After Xcode version 7.x, Apple started shipping the `Xcode.app` in a `.xip`
archive. This makes the SDK less-trivial to extract on non-macOS machines. One
approach (tested on Debian Buster) is outlined below:
```bash
# Install/clone tools needed for extracting Xcode.app
apt install cpio
git clone https://github.com/digibyte-core/apple-sdk-tools.git
# Unpack Xcode_12.2.xip and place the resulting Xcode.app in your current
# working directory
python3 apple-sdk-tools/extract_xcode.py -f Xcode_12.2.xip | cpio -d -i
```
On macOS the process is more straightforward:
```bash
xip -x Xcode_12.2.xip
```
### Step 2: Generating `Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz` from `Xcode.app`
To generate `Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz`, run
the script [`gen-sdk`](./gen-sdk) with the path to `Xcode.app` (extracted in the
previous stage) as the first argument.
```bash
# Generate a Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz from
# the supplied Xcode.app
./contrib/macdeploy/gen-sdk '/path/to/Xcode.app'
```
## Deterministic macOS DMG Notes
Working macOS DMGs are created in Linux by combining a recent `clang`, the Apple
`binutils` (`ld`, `ar`, etc) and DMG authoring tools.
Apple uses `clang` extensively for development and has upstreamed the necessary
functionality so that a vanilla clang can take advantage. It supports the use of `-F`,
`-target`, `-mmacosx-version-min`, and `-isysroot`, which are all necessary when
building for macOS.
Apple's version of `binutils` (called `cctools`) contains lots of functionality missing in the
FSF's `binutils`. In addition to extra linker options for frameworks and sysroots, severalother tools are needed as well such as `install_name_tool`, `lipo`, and `nmedit`. These
do not build under Linux, so they have been patched to do so. The work here was used as
a starting point: [mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4).
In order to build a working toolchain, the following source packages are needed from
Apple: `cctools`, `dyld`, and `ld64`.
These tools inject timestamps by default, which produce non-deterministic binaries. The
`ZERO_AR_DATE` environment variable is used to disable that.
This version of `cctools` has been patched to use the current version of `clang`'s headers
and its `libLTO.so` rather than those from `llvmgcc`, as it was originally done in `toolchain4`.
To complicate things further, all builds must target an Apple SDK. These SDKs are free to
download, but not redistributable. To obtain it, register for an Apple Developer Account,
then download [Xcode_12.1](https://download.developer.apple.com/Developer_Tools/Xcode_12.1/Xcode_12.1.xip).
This file is many gigabytes in size, but most (but not all) of what we need is
contained only in a single directory:
```bash
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
```
See the SDK Extraction notes above for how to obtain it.
The Guix process build 2 sets of files: Linux tools, then Apple binaries which are
created using these tools. The build process has been designed to avoid including the
SDK's files in Guix's outputs. All interim tarballs are fully deterministic and may be freely
redistributed.
[`xorrisofs`](https://www.gnu.org/software/xorriso/) is used to create the DMG.
`xorrisofs` cannot compress DMGs, so afterwards, the DMG tool from the
`libdmg-hfsplus` project is used to compress it. There are several bugs in this
tool and its maintainer has seemingly abandoned the project.
The DMG tool has the ability to create DMGs from scratch as well, but this functionality is
broken. Only the compression feature is currently used. Ideally, the creation could be fixed
and `xorrisofs` would no longer be necessary.
Background images and other features can be added to DMG files by inserting a
`.DS_Store` during creation.
As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a requirement in
order to satisfy the new Gatekeeper requirements. Because this private key cannot be
shared, we'll have to be a bit creative in order for the build process to remain somewhat
deterministic. Here's how it works:
- Builders use Guix to create an unsigned release. This outputs an unsigned DMG which
users may choose to bless and run. It also outputs an unsigned app structure in the form
of a tarball, which also contains all of the tools that have been previously (deterministically)
built in order to create a final DMG.
- The Apple keyholder uses this unsigned app to create a detached signature, using the
script that is also included there. Detached signatures are available from this [repository](https://github.com/digibyte-core/digibyte-detached-sigs).
- Builders feed the unsigned app + detached signature back into Guix. It uses the
pre-built tools to recombine the pieces into a deterministic DMG.

34
contrib/macdeploy/background.svg Normal file
View file

@ -0,0 +1,34 @@
<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 20010904//EN"
"http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
<svg version="1.0" xmlns="http://www.w3.org/2000/svg" width="1000pt" height="640pt" viewBox="0 0 1000 640" preserveAspectRatio="xMidYMid meet">
<!-- kate: space-indent off;
Copyright (c) 2015 The DigiByte Core developers
Distributed under the MIT software license, see the accompanying
file COPYING or http://www.opensource.org/licenses/mit-license.php.
-->
<style type="text/css"><![CDATA[
text {
font-family: "Tuffy";
font-size: 86px;
fill: gray;
text-anchor: middle;
}
]]></style>
<defs>
<linearGradient id="gradient" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:rgb(239,239,239);stop-opacity:1" />
<stop offset="33%" style="stop-color:rgb(239,239,239);stop-opacity:1" />
<stop offset="80%" style="stop-color:rgb(205,205,205);stop-opacity:1" />
<stop offset="100%" style="stop-color:rgb(204,204,204);stop-opacity:1" />
</linearGradient>
</defs>
<rect width="1000" height="640" style="fill:url(#gradient);stroke-width:0" />
<g transform="translate(500,0) scale(0.9, 1)">
<text x="0" y="114">PACKAGE_NAME</text>
</g>
<g transform="translate(0.000000,640.000000) scale(0.100000,-0.100000)"
fill="#000000" stroke="none">
<path d="M4995 3705 c-24 -23 -25 -29 -25 -165 l0 -140 -306 0 -306 0 -29 -29 c-29 -29 -29 -31 -29 -141 0 -110 0 -112 29 -141 l29 -29 306 0 306 0 0 -140 c0 -136 1 -142 25 -165 16 -17 35 -25 57 -25 29 0 72 32 306 226 180 149 274 233 278 250 13 53 -2 70 -278 299 -235 194 -277 225 -306 225 -22 0 -41 -8 -57 -25z" fixlter="url(#glow)"/>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 1.7 KiB

28
contrib/macdeploy/detached-sig-apply.sh Executable file
View file

@ -0,0 +1,28 @@
#!/bin/sh
# Copyright (c) 2014-2015 The Bitcoin Core developers
# Copyright (c) 2014-2019 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
set -e
UNSIGNED="$1"
SIGNATURE="$2"
ROOTDIR=dist
OUTDIR=signed-app
SIGNAPPLE=signapple
if [ -z "$UNSIGNED" ]; then
echo "usage: $0 <unsigned app> <signature>"
exit 1
fi
if [ -z "$SIGNATURE" ]; then
echo "usage: $0 <unsigned app> <signature>"
exit 1
fi
${SIGNAPPLE} apply ${UNSIGNED} ${SIGNATURE}
mv ${ROOTDIR} ${OUTDIR}
echo "Signed: ${OUTDIR}"

30
contrib/macdeploy/detached-sig-create.sh Executable file
View file

@ -0,0 +1,30 @@
#!/bin/sh
# Copyright (c) 2014-2015 The Bitcoin Core developers
# Copyright (c) 2014-2019 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
set -e
ROOTDIR=dist
BUNDLE="${ROOTDIR}/DigiByte-Qt.app"
SIGNAPPLE=signapple
TEMPDIR=sign.temp
OUT=signature-osx.tar.gz
OUTROOT=osx/dist
if [ -z "$1" ]; then
echo "usage: $0 <signapple args>"
echo "example: $0 <path to key>"
exit 1
fi
rm -rf ${TEMPDIR}
mkdir -p ${TEMPDIR}
${SIGNAPPLE} sign -f --detach "${TEMPDIR}/${OUTROOT}" "$@" "${BUNDLE}"
tar -C "${TEMPDIR}" -czf "${OUT}" .
rm -rf "${TEMPDIR}"
echo "Created ${OUT}"

34
contrib/macdeploy/extract-osx-sdk.sh Executable file
View file

@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Copyright (c) 2016 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
set -e
INPUTFILE="Xcode_7.3.1.dmg"
HFSFILENAME="5.hfs"
SDKDIR="Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk"
7z x "${INPUTFILE}" "${HFSFILENAME}"
SDKNAME="$(basename "${SDKDIR}")"
SDKDIRINODE=$(ifind -n "${SDKDIR}" "${HFSFILENAME}")
fls "${HFSFILENAME}" -rpF ${SDKDIRINODE} |
while read type inode filename; do
inode="${inode::-1}"
if [ "${filename:0:14}" = "usr/share/man/" ]; then
continue
fi
filename="${SDKNAME}/$filename"
echo "Extracting $filename ..."
mkdir -p "$(dirname "$filename")"
if [ "$type" = "l/l" ]; then
ln -s "$(icat "${HFSFILENAME}" $inode)" "$filename"
else
icat "${HFSFILENAME}" $inode >"$filename"
fi
done
echo "Building ${SDKNAME}.tar.gz ..."
MTIME="$(istat "${HFSFILENAME}" "${SDKDIRINODE}" | perl -nle 'm/Content Modified:\s+(.*?)\s\(/ && print $1')"
find "${SDKNAME}" | sort | tar --no-recursion --mtime="${MTIME}" --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > "${SDKNAME}.tar.gz"
echo 'All done!'

94
contrib/macdeploy/gen-sdk Executable file
View file

@ -0,0 +1,94 @@
#!/usr/bin/env python3
import argparse
import plistlib
import pathlib
import sys
import tarfile
import gzip
import os
import contextlib
@contextlib.contextmanager
def cd(path):
"""Context manager that restores PWD even if an exception was raised."""
old_pwd = os.getcwd()
os.chdir(str(path))
try:
yield
finally:
os.chdir(old_pwd)
def run():
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdktgz', required=False)
args = parser.parse_args()
xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
with xcode_app_plist.open('rb') as fp:
pl = plistlib.load(fp)
xcode_version = pl['CFBundleShortVersionString']
xcode_build_id = pl['ProductBuildVersion']
print("Found Xcode (version: {xcode_version}, build id: {xcode_build_id})".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id))
sdk_dir = xcode_app.joinpath("Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk")
sdk_plist = sdk_dir.joinpath("System/Library/CoreServices/SystemVersion.plist")
with sdk_plist.open('rb') as fp:
pl = plistlib.load(fp)
sdk_version = pl['ProductVersion']
sdk_build_id = pl['ProductBuildVersion']
print("Found MacOSX SDK (version: {sdk_version}, build id: {sdk_build_id})".format(sdk_version=sdk_version, sdk_build_id=sdk_build_id))
out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
xcode_libcxx_dir = xcode_app.joinpath("Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1")
assert xcode_libcxx_dir.is_dir()
if args.out_sdktgz:
out_sdktgz_path = pathlib.Path(args.out_sdktgz_path)
else:
# Construct our own out_sdktgz if not specified on the command line
out_sdktgz_path = pathlib.Path("./{}.tar.gz".format(out_name))
def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
"""Add all files in dir_to_add to tarfp, but prepent MEMBERPREFIX to the files'
names
e.g. if the only file under /root/bazdir is /root/bazdir/qux, invoking:
tarfp_add_with_base_change(tarfp, "foo/bar", "/root/bazdir")
would result in the following members being added to tarfp:
foo/bar/ -> corresponding to /root/bazdir
foo/bar/qux -> corresponding to /root/bazdir/qux
"""
def change_tarinfo_base(tarinfo):
if tarinfo.name and tarinfo.name.startswith("./"):
tarinfo.name = str(pathlib.Path(alt_base_dir, tarinfo.name))
if tarinfo.linkname and tarinfo.linkname.startswith("./"):
tarinfo.linkname = str(pathlib.Path(alt_base_dir, tarinfo.linkname))
return tarinfo
with cd(dir_to_add):
tarfp.add(".", recursive=True, filter=change_tarinfo_base)
print("Creating output .tar.gz file...")
with out_sdktgz_path.open("wb") as fp:
with gzip.GzipFile(fileobj=fp, compresslevel=9, mtime=0) as gzf:
with tarfile.open(mode="w", fileobj=gzf) as tarfp:
print("Adding MacOSX SDK {} files...".format(sdk_version))
tarfp_add_with_base_change(tarfp, sdk_dir, out_name)
print("Adding libc++ headers...")
tarfp_add_with_base_change(tarfp, xcode_libcxx_dir, "{}/usr/include/c++/v1".format(out_name))
print("Done! Find the resulting gzipped tarball at:")
print(out_sdktgz_path.resolve())
if __name__ == '__main__':
run()

594
contrib/macdeploy/macdeployqtplus Executable file
View file

@ -0,0 +1,594 @@
#!/usr/bin/env python3
#
# Copyright (C) 2011 Patrick "p2k" Schneider <me@p2k-network.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import sys, re, os, shutil, stat, os.path
from argparse import ArgumentParser
from ds_store import DSStore
from mac_alias import Alias
from pathlib import Path
from subprocess import PIPE, run
from typing import List, Optional
# This is ported from the original macdeployqt with modifications
class FrameworkInfo(object):
def __init__(self):
self.frameworkDirectory = ""
self.frameworkName = ""
self.frameworkPath = ""
self.binaryDirectory = ""
self.binaryName = ""
self.binaryPath = ""
self.version = ""
self.installName = ""
self.deployedInstallName = ""
self.sourceFilePath = ""
self.destinationDirectory = ""
self.sourceResourcesDirectory = ""
self.sourceVersionContentsDirectory = ""
self.sourceContentsDirectory = ""
self.destinationResourcesDirectory = ""
self.destinationVersionContentsDirectory = ""
def __eq__(self, other):
if self.__class__ == other.__class__:
return self.__dict__ == other.__dict__
else:
return False
def __str__(self):
return f""" Framework name: {self.frameworkName}
Framework directory: {self.frameworkDirectory}
Framework path: {self.frameworkPath}
Binary name: {self.binaryName}
Binary directory: {self.binaryDirectory}
Binary path: {self.binaryPath}
Version: {self.version}
Install name: {self.installName}
Deployed install name: {self.deployedInstallName}
Source file Path: {self.sourceFilePath}
Deployed Directory (relative to bundle): {self.destinationDirectory}
"""
def isDylib(self):
return self.frameworkName.endswith(".dylib")
def isQtFramework(self):
if self.isDylib():
return self.frameworkName.startswith("libQt")
else:
return self.frameworkName.startswith("Qt")
reOLine = re.compile(r'^(.+) \(compatibility version [0-9.]+, current version [0-9.]+\)$')
bundleFrameworkDirectory = "Contents/Frameworks"
bundleBinaryDirectory = "Contents/MacOS"
@classmethod
def fromOtoolLibraryLine(cls, line: str) -> Optional['FrameworkInfo']:
# Note: line must be trimmed
if line == "":
return None
# Don't deploy system libraries
if line.startswith("/System/Library/") or line.startswith("@executable_path") or line.startswith("/usr/lib/"):
return None
m = cls.reOLine.match(line)
if m is None:
raise RuntimeError(f"otool line could not be parsed: {line}")
path = m.group(1)
info = cls()
info.sourceFilePath = path
info.installName = path
if path.endswith(".dylib"):
dirname, filename = os.path.split(path)
info.frameworkName = filename
info.frameworkDirectory = dirname
info.frameworkPath = path
info.binaryDirectory = dirname
info.binaryName = filename
info.binaryPath = path
info.version = "-"
info.installName = path
info.deployedInstallName = f"@executable_path/../Frameworks/{info.binaryName}"
info.sourceFilePath = path
info.destinationDirectory = cls.bundleFrameworkDirectory
else:
parts = path.split("/")
i = 0
# Search for the .framework directory
for part in parts:
if part.endswith(".framework"):
break
i += 1
if i == len(parts):
raise RuntimeError(f"Could not find .framework or .dylib in otool line: {line}")
info.frameworkName = parts[i]
info.frameworkDirectory = "/".join(parts[:i])
info.frameworkPath = os.path.join(info.frameworkDirectory, info.frameworkName)
info.binaryName = parts[i+3]
info.binaryDirectory = "/".join(parts[i+1:i+3])
info.binaryPath = os.path.join(info.binaryDirectory, info.binaryName)
info.version = parts[i+2]
info.deployedInstallName = f"@executable_path/../Frameworks/{os.path.join(info.frameworkName, info.binaryPath)}"
info.destinationDirectory = os.path.join(cls.bundleFrameworkDirectory, info.frameworkName, info.binaryDirectory)
info.sourceResourcesDirectory = os.path.join(info.frameworkPath, "Resources")
info.sourceContentsDirectory = os.path.join(info.frameworkPath, "Contents")
info.sourceVersionContentsDirectory = os.path.join(info.frameworkPath, "Versions", info.version, "Contents")
info.destinationResourcesDirectory = os.path.join(cls.bundleFrameworkDirectory, info.frameworkName, "Resources")
info.destinationVersionContentsDirectory = os.path.join(cls.bundleFrameworkDirectory, info.frameworkName, "Versions", info.version, "Contents")
return info
class ApplicationBundleInfo(object):
def __init__(self, path: str):
self.path = path
# for backwards compatibility reasons, this must remain as DigiByte-Qt
self.binaryPath = os.path.join(path, "Contents", "MacOS", "DigiByte-Qt")
if not os.path.exists(self.binaryPath):
raise RuntimeError(f"Could not find bundle binary for {path}")
self.resourcesPath = os.path.join(path, "Contents", "Resources")
self.pluginPath = os.path.join(path, "Contents", "PlugIns")
class DeploymentInfo(object):
def __init__(self):
self.qtPath = None
self.pluginPath = None
self.deployedFrameworks = []
def detectQtPath(self, frameworkDirectory: str):
parentDir = os.path.dirname(frameworkDirectory)
if os.path.exists(os.path.join(parentDir, "translations")):
# Classic layout, e.g. "/usr/local/Trolltech/Qt-4.x.x"
self.qtPath = parentDir
else:
self.qtPath = os.getenv("QTDIR", None)
if self.qtPath is not None:
pluginPath = os.path.join(self.qtPath, "plugins")
if os.path.exists(pluginPath):
self.pluginPath = pluginPath
def usesFramework(self, name: str) -> bool:
for framework in self.deployedFrameworks:
if framework.endswith(".framework"):
if framework.startswith(f"{name}."):
return True
elif framework.endswith(".dylib"):
if framework.startswith(f"lib{name}."):
return True
return False
def getFrameworks(binaryPath: str, verbose: int) -> List[FrameworkInfo]:
if verbose:
print(f"Inspecting with otool: {binaryPath}")
otoolbin=os.getenv("OTOOL", "otool")
otool = run([otoolbin, "-L", binaryPath], stdout=PIPE, stderr=PIPE, universal_newlines=True)
if otool.returncode != 0:
sys.stderr.write(otool.stderr)
sys.stderr.flush()
raise RuntimeError(f"otool failed with return code {otool.returncode}")
otoolLines = otool.stdout.split("\n")
otoolLines.pop(0) # First line is the inspected binary
if ".framework" in binaryPath or binaryPath.endswith(".dylib"):
otoolLines.pop(0) # Frameworks and dylibs list themselves as a dependency.
libraries = []
for line in otoolLines:
line = line.replace("@loader_path", os.path.dirname(binaryPath))
info = FrameworkInfo.fromOtoolLibraryLine(line.strip())
if info is not None:
if verbose:
print("Found framework:")
print(info)
libraries.append(info)
return libraries
def runInstallNameTool(action: str, *args):
installnametoolbin=os.getenv("INSTALLNAMETOOL", "install_name_tool")
run([installnametoolbin, "-"+action] + list(args), check=True)
def changeInstallName(oldName: str, newName: str, binaryPath: str, verbose: int):
if verbose:
print("Using install_name_tool:")
print(" in", binaryPath)
print(" change reference", oldName)
print(" to", newName)
runInstallNameTool("change", oldName, newName, binaryPath)
def changeIdentification(id: str, binaryPath: str, verbose: int):
if verbose:
print("Using install_name_tool:")
print(" change identification in", binaryPath)
print(" to", id)
runInstallNameTool("id", id, binaryPath)
def runStrip(binaryPath: str, verbose: int):
stripbin=os.getenv("STRIP", "strip")
if verbose:
print("Using strip:")
print(" stripped", binaryPath)
run([stripbin, "-x", binaryPath], check=True)
def copyFramework(framework: FrameworkInfo, path: str, verbose: int) -> Optional[str]:
if framework.sourceFilePath.startswith("Qt"):
#standard place for Nokia Qt installer's frameworks
fromPath = f"/Library/Frameworks/{framework.sourceFilePath}"
else:
fromPath = framework.sourceFilePath
toDir = os.path.join(path, framework.destinationDirectory)
toPath = os.path.join(toDir, framework.binaryName)
if not os.path.exists(fromPath):
raise RuntimeError(f"No file at {fromPath}")
if os.path.exists(toPath):
return None # Already there
if not os.path.exists(toDir):
os.makedirs(toDir)
shutil.copy2(fromPath, toPath)
if verbose:
print("Copied:", fromPath)
print(" to:", toPath)
permissions = os.stat(toPath)
if not permissions.st_mode & stat.S_IWRITE:
os.chmod(toPath, permissions.st_mode | stat.S_IWRITE)
if not framework.isDylib(): # Copy resources for real frameworks
linkfrom = os.path.join(path, "Contents","Frameworks", framework.frameworkName, "Versions", "Current")
linkto = framework.version
if not os.path.exists(linkfrom):
os.symlink(linkto, linkfrom)
print("Linked:", linkfrom, "->", linkto)
fromResourcesDir = framework.sourceResourcesDirectory
if os.path.exists(fromResourcesDir):
toResourcesDir = os.path.join(path, framework.destinationResourcesDirectory)
shutil.copytree(fromResourcesDir, toResourcesDir, symlinks=True)
if verbose:
print("Copied resources:", fromResourcesDir)
print(" to:", toResourcesDir)
fromContentsDir = framework.sourceVersionContentsDirectory
if not os.path.exists(fromContentsDir):
fromContentsDir = framework.sourceContentsDirectory
if os.path.exists(fromContentsDir):
toContentsDir = os.path.join(path, framework.destinationVersionContentsDirectory)
shutil.copytree(fromContentsDir, toContentsDir, symlinks=True)
if verbose:
print("Copied Contents:", fromContentsDir)
print(" to:", toContentsDir)
return toPath
def deployFrameworks(frameworks: List[FrameworkInfo], bundlePath: str, binaryPath: str, strip: bool, verbose: int, deploymentInfo: Optional[DeploymentInfo] = None) -> DeploymentInfo:
if deploymentInfo is None:
deploymentInfo = DeploymentInfo()
while len(frameworks) > 0:
framework = frameworks.pop(0)
deploymentInfo.deployedFrameworks.append(framework.frameworkName)
print("Processing", framework.frameworkName, "...")
# Get the Qt path from one of the Qt frameworks
if deploymentInfo.qtPath is None and framework.isQtFramework():
deploymentInfo.detectQtPath(framework.frameworkDirectory)
if framework.installName.startswith("@executable_path") or framework.installName.startswith(bundlePath):
print(framework.frameworkName, "already deployed, skipping.")
continue
# install_name_tool the new id into the binary
changeInstallName(framework.installName, framework.deployedInstallName, binaryPath, verbose)
# Copy framework to app bundle.
deployedBinaryPath = copyFramework(framework, bundlePath, verbose)
# Skip the rest if already was deployed.
if deployedBinaryPath is None:
continue
if strip:
runStrip(deployedBinaryPath, verbose)
# install_name_tool it a new id.
changeIdentification(framework.deployedInstallName, deployedBinaryPath, verbose)
# Check for framework dependencies
dependencies = getFrameworks(deployedBinaryPath, verbose)
for dependency in dependencies:
changeInstallName(dependency.installName, dependency.deployedInstallName, deployedBinaryPath, verbose)
# Deploy framework if necessary.
if dependency.frameworkName not in deploymentInfo.deployedFrameworks and dependency not in frameworks:
frameworks.append(dependency)
return deploymentInfo
def deployFrameworksForAppBundle(applicationBundle: ApplicationBundleInfo, strip: bool, verbose: int) -> DeploymentInfo:
frameworks = getFrameworks(applicationBundle.binaryPath, verbose)
if len(frameworks) == 0:
print(f"Warning: Could not find any external frameworks to deploy in {applicationBundle.path}.")
return DeploymentInfo()
else:
return deployFrameworks(frameworks, applicationBundle.path, applicationBundle.binaryPath, strip, verbose)
def deployPlugins(appBundleInfo: ApplicationBundleInfo, deploymentInfo: DeploymentInfo, strip: bool, verbose: int):
plugins = []
if deploymentInfo.pluginPath is None:
return
for dirpath, dirnames, filenames in os.walk(deploymentInfo.pluginPath):
pluginDirectory = os.path.relpath(dirpath, deploymentInfo.pluginPath)
if pluginDirectory not in ['styles', 'platforms']:
continue
for pluginName in filenames:
pluginPath = os.path.join(pluginDirectory, pluginName)
if pluginName.split('.')[0] not in ['libqminimal', 'libqcocoa', 'libqmacstyle']:
continue
plugins.append((pluginDirectory, pluginName))
for pluginDirectory, pluginName in plugins:
print("Processing plugin", os.path.join(pluginDirectory, pluginName), "...")
sourcePath = os.path.join(deploymentInfo.pluginPath, pluginDirectory, pluginName)
destinationDirectory = os.path.join(appBundleInfo.pluginPath, pluginDirectory)
if not os.path.exists(destinationDirectory):
os.makedirs(destinationDirectory)
destinationPath = os.path.join(destinationDirectory, pluginName)
shutil.copy2(sourcePath, destinationPath)
if verbose:
print("Copied:", sourcePath)
print(" to:", destinationPath)
if strip:
runStrip(destinationPath, verbose)
dependencies = getFrameworks(destinationPath, verbose)
for dependency in dependencies:
changeInstallName(dependency.installName, dependency.deployedInstallName, destinationPath, verbose)
# Deploy framework if necessary.
if dependency.frameworkName not in deploymentInfo.deployedFrameworks:
deployFrameworks([dependency], appBundleInfo.path, destinationPath, strip, verbose, deploymentInfo)
ap = ArgumentParser(description="""Improved version of macdeployqt.
Outputs a ready-to-deploy app in a folder "dist" and optionally wraps it in a .dmg file.
Note, that the "dist" folder will be deleted before deploying on each run.
Optionally, Qt translation files (.qm) can be added to the bundle.""")
ap.add_argument("app_bundle", nargs=1, metavar="app-bundle", help="application bundle to be deployed")
ap.add_argument("appname", nargs=1, metavar="appname", help="name of the app being deployed")
ap.add_argument("-verbose", nargs="?", const=True, help="Output additional debugging information")
ap.add_argument("-no-plugins", dest="plugins", action="store_false", default=True, help="skip plugin deployment")
ap.add_argument("-no-strip", dest="strip", action="store_false", default=True, help="don't run 'strip' on the binaries")
ap.add_argument("-dmg", nargs="?", const="", metavar="basename", help="create a .dmg disk image")
ap.add_argument("-translations-dir", nargs=1, metavar="path", default=None, help="Path to Qt's translations. Base translations will automatically be added to the bundle's resources.")
config = ap.parse_args()
verbose = config.verbose
# ------------------------------------------------
app_bundle = config.app_bundle[0]
appname = config.appname[0]
if not os.path.exists(app_bundle):
sys.stderr.write(f"Error: Could not find app bundle \"{app_bundle}\"\n")
sys.exit(1)
# ------------------------------------------------
if os.path.exists("dist"):
print("+ Removing existing dist folder +")
shutil.rmtree("dist")
if os.path.exists(appname + ".dmg"):
print("+ Removing existing DMG +")
os.unlink(appname + ".dmg")
if os.path.exists(appname + ".temp.dmg"):
os.unlink(appname + ".temp.dmg")
# ------------------------------------------------
target = os.path.join("dist", "DigiByte-Qt.app")
print("+ Copying source bundle +")
if verbose:
print(app_bundle, "->", target)
os.mkdir("dist")
shutil.copytree(app_bundle, target, symlinks=True)
applicationBundle = ApplicationBundleInfo(target)
# ------------------------------------------------
print("+ Deploying frameworks +")
try:
deploymentInfo = deployFrameworksForAppBundle(applicationBundle, config.strip, verbose)
if deploymentInfo.qtPath is None:
deploymentInfo.qtPath = os.getenv("QTDIR", None)
if deploymentInfo.qtPath is None:
sys.stderr.write("Warning: Could not detect Qt's path, skipping plugin deployment!\n")
config.plugins = False
except RuntimeError as e:
sys.stderr.write(f"Error: {str(e)}\n")
sys.exit(1)
# ------------------------------------------------
if config.plugins:
print("+ Deploying plugins +")
try:
deployPlugins(applicationBundle, deploymentInfo, config.strip, verbose)
except RuntimeError as e:
sys.stderr.write(f"Error: {str(e)}\n")
sys.exit(1)
# ------------------------------------------------
if config.translations_dir:
if not Path(config.translations_dir[0]).exists():
sys.stderr.write(f"Error: Could not find translation dir \"{config.translations_dir[0]}\"\n")
sys.exit(1)
print("+ Adding Qt translations +")
translations = Path(config.translations_dir[0])
regex = re.compile('qt_[a-z]*(.qm|_[A-Z]*.qm)')
lang_files = [x for x in translations.iterdir() if regex.match(x.name)]
for file in lang_files:
if verbose:
print(file.as_posix(), "->", os.path.join(applicationBundle.resourcesPath, file.name))
shutil.copy2(file.as_posix(), os.path.join(applicationBundle.resourcesPath, file.name))
# ------------------------------------------------
print("+ Installing qt.conf +")
qt_conf="""[Paths]
Translations=Resources
Plugins=PlugIns
"""
with open(os.path.join(applicationBundle.resourcesPath, "qt.conf"), "wb") as f:
f.write(qt_conf.encode())
# ------------------------------------------------
print("+ Generating .DS_Store +")
output_file = os.path.join("dist", ".DS_Store")
ds = DSStore.open(output_file, 'w+')
ds['.']['bwsp'] = {
'WindowBounds': '{{300, 280}, {500, 343}}',
'PreviewPaneVisibility': False,
}
icvp = {
'gridOffsetX': 0.0,
'textSize': 12.0,
'viewOptionsVersion': 1,
'backgroundImageAlias': b'\x00\x00\x00\x00\x02\x1e\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xd1\x94\\\xb0H+\x00\x05\x00\x00\x00\x98\x0fbackground.tiff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x99\xd19\xb0\xf8\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\x00\x00\r\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0b.background\x00\x00\x10\x00\x08\x00\x00\xd1\x94\\\xb0\x00\x00\x00\x11\x00\x08\x00\x00\xd19\xb0\xf8\x00\x00\x00\x01\x00\x04\x00\x00\x00\x98\x00\x0e\x00 \x00\x0f\x00b\x00a\x00c\x00k\x00g\x00r\x00o\x00u\x00n\x00d\x00.\x00t\x00i\x00f\x00f\x00\x0f\x00\x02\x00\x00\x00\x12\x00\x1c/.background/background.tiff\x00\x14\x01\x06\x00\x00\x00\x00\x01\x06\x00\x02\x00\x00\x0cMacintosh HD\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xce\x97\xab\xc3H+\x00\x00\x01\x88[\x88\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02u\xab\x8d\xd1\x94\\\xb0devrddsk\xff\xff\xff\xff\x00\x00\t \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x07bitcoin\x00\x00\x10\x00\x08\x00\x00\xce\x97\xab\xc3\x00\x00\x00\x11\x00\x08\x00\x00\xd1\x94\\\xb0\x00\x00\x00\x01\x00\x14\x01\x88[\x88\x00\x16\xa9\t\x00\x08\xfaR\x00\x08\xfaQ\x00\x02d\x8e\x00\x0e\x00\x02\x00\x00\x00\x0f\x00\x1a\x00\x0c\x00M\x00a\x00c\x00i\x00n\x00t\x00o\x00s\x00h\x00 \x00H\x00D\x00\x13\x00\x01/\x00\x00\x15\x00\x02\x00\x14\xff\xff\x00\x00\xff\xff\x00\x00',
'backgroundColorBlue': 1.0,
'iconSize': 96.0,
'backgroundColorGreen': 1.0,
'arrangeBy': 'none',
'showIconPreview': True,
'gridSpacing': 100.0,
'gridOffsetY': 0.0,
'showItemInfo': False,
'labelOnBottom': True,
'backgroundType': 2,
'backgroundColorRed': 1.0
}
alias = Alias().from_bytes(icvp['backgroundImageAlias'])
alias.volume.name = appname
alias.volume.posix_path = '/Volumes/' + appname
icvp['backgroundImageAlias'] = alias.to_bytes()
ds['.']['icvp'] = icvp
ds['.']['vSrn'] = ('long', 1)
ds['Applications']['Iloc'] = (370, 156)
ds['DigiByte-Qt.app']['Iloc'] = (128, 156)
ds.flush()
ds.close()
# ------------------------------------------------
if config.dmg is not None:
print("+ Preparing .dmg disk image +")
if verbose:
print("Determining size of \"dist\"...")
size = 0
for path, dirs, files in os.walk("dist"):
for file in files:
size += os.path.getsize(os.path.join(path, file))
size += int(size * 0.15)
if verbose:
print("Creating temp image for modification...")
tempname: str = appname + ".temp.dmg"
run(["hdiutil", "create", tempname, "-srcfolder", "dist", "-format", "UDRW", "-size", str(size), "-volname", appname], check=True, universal_newlines=True)
if verbose:
print("Attaching temp image...")
output = run(["hdiutil", "attach", tempname, "-readwrite"], check=True, universal_newlines=True, stdout=PIPE).stdout
m = re.search(r"/Volumes/(.+$)", output)
disk_root = m.group(0)
print("+ Applying fancy settings +")
bg_path = os.path.join(disk_root, ".background", os.path.basename('background.tiff'))
os.mkdir(os.path.dirname(bg_path))
if verbose:
print('background.tiff', "->", bg_path)
shutil.copy2('background.tiff', bg_path)
os.symlink("/Applications", os.path.join(disk_root, "Applications"))
print("+ Finalizing .dmg disk image +")
run(["hdiutil", "detach", f"/Volumes/{appname}"], universal_newlines=True)
run(["hdiutil", "convert", tempname, "-format", "UDZO", "-o", appname, "-imagekey", "zlib-level=9"], check=True, universal_newlines=True)
os.unlink(tempname)
# ------------------------------------------------
print("+ Done +")
sys.exit(0)

25
contrib/message-capture/message-capture-docs.md Normal file
View file

@ -0,0 +1,25 @@
# Per-Peer Message Capture
## Purpose
This feature allows for message capture on a per-peer basis. It answers the simple question: "Can I see what messages my node is sending and receiving?"
## Usage and Functionality
* Run `digibyted` with the `-capturemessages` option.
* Look in the `message_capture` folder in your datadir.
* Typically this will be `~/.digibyte/message_capture`.
* See that there are many folders inside, one for each peer names with its IP address and port.
* Inside each peer's folder there are two `.dat` files: one is for received messages (`msgs_recv.dat`) and the other is for sent messages (`msgs_sent.dat`).
* Run `contrib/message-capture/message-capture-parser.py` with the proper arguments.
* See the `-h` option for help.
* To see all messages, both sent and received, for all peers use:
```
./contrib/message-capture/message-capture-parser.py -o out.json \
~/.digibyte/message_capture/**/*.dat
```
* Note: The messages in the given `.dat` files will be interleaved in chronological order. So, giving both received and sent `.dat` files (as above with `*.dat`) will result in all messages being interleaved in chronological order.
* If an output file is not provided (i.e. the `-o` option is not used), then the output prints to `stdout`.
* View the resulting output.
* The output file is `JSON` formatted.
* Suggestion: use `jq` to view the output, with `jq . out.json`

214
contrib/message-capture/message-capture-parser.py Executable file
View file

@ -0,0 +1,214 @@
#!/usr/bin/env python3
# Copyright (c) 2020 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""Parse message capture binary files. To be used in conjunction with -capturemessages."""
import argparse
import os
import shutil
import sys
from io import BytesIO
import json
from pathlib import Path
from typing import Any, List, Optional
sys.path.append(os.path.join(os.path.dirname(__file__), '../../test/functional'))
from test_framework.messages import ser_uint256 # noqa: E402
from test_framework.p2p import MESSAGEMAP # noqa: E402
TIME_SIZE = 8
LENGTH_SIZE = 4
MSGTYPE_SIZE = 12
# The test framework classes stores hashes as large ints in many cases.
# These are variables of type uint256 in core.
# There isn't a way to distinguish between a large int and a large int that is actually a blob of bytes.
# As such, they are itemized here.
# Any variables with these names that are of type int are actually uint256 variables.
# (These can be easily found by looking for calls to deser_uint256, deser_uint256_vector, and uint256_from_str in messages.py)
HASH_INTS = [
"blockhash",
"block_hash",
"hash",
"hashMerkleRoot",
"hashPrevBlock",
"hashstop",
"prev_header",
"sha256",
"stop_hash",
]
HASH_INT_VECTORS = [
"hashes",
"headers",
"vHave",
"vHash",
]
class ProgressBar:
def __init__(self, total: float):
self.total = total
self.running = 0
def set_progress(self, progress: float):
cols = shutil.get_terminal_size()[0]
if cols <= 12:
return
max_blocks = cols - 9
num_blocks = int(max_blocks * progress)
print('\r[ {}{} ] {:3.0f}%'
.format('#' * num_blocks,
' ' * (max_blocks - num_blocks),
progress * 100),
end ='')
def update(self, more: float):
self.running += more
self.set_progress(self.running / self.total)
def to_jsonable(obj: Any) -> Any:
if hasattr(obj, "__dict__"):
return obj.__dict__
elif hasattr(obj, "__slots__"):
ret = {} # type: Any
for slot in obj.__slots__:
val = getattr(obj, slot, None)
if slot in HASH_INTS and isinstance(val, int):
ret[slot] = ser_uint256(val).hex()
elif slot in HASH_INT_VECTORS and isinstance(val[0], int):
ret[slot] = [ser_uint256(a).hex() for a in val]
else:
ret[slot] = to_jsonable(val)
return ret
elif isinstance(obj, list):
return [to_jsonable(a) for a in obj]
elif isinstance(obj, bytes):
return obj.hex()
else:
return obj
def process_file(path: str, messages: List[Any], recv: bool, progress_bar: Optional[ProgressBar]) -> None:
with open(path, 'rb') as f_in:
if progress_bar:
bytes_read = 0
while True:
if progress_bar:
# Update progress bar
diff = f_in.tell() - bytes_read - 1
progress_bar.update(diff)
bytes_read = f_in.tell() - 1
# Read the Header
tmp_header_raw = f_in.read(TIME_SIZE + LENGTH_SIZE + MSGTYPE_SIZE)
if not tmp_header_raw:
break
tmp_header = BytesIO(tmp_header_raw)
time = int.from_bytes(tmp_header.read(TIME_SIZE), "little") # type: int
msgtype = tmp_header.read(MSGTYPE_SIZE).split(b'\x00', 1)[0] # type: bytes
length = int.from_bytes(tmp_header.read(LENGTH_SIZE), "little") # type: int
# Start converting the message to a dictionary
msg_dict = {}
msg_dict["direction"] = "recv" if recv else "sent"
msg_dict["time"] = time
msg_dict["size"] = length # "size" is less readable here, but more readable in the output
msg_ser = BytesIO(f_in.read(length))
# Determine message type
if msgtype not in MESSAGEMAP:
# Unrecognized message type
try:
msgtype_tmp = msgtype.decode()
if not msgtype_tmp.isprintable():
raise UnicodeDecodeError
msg_dict["msgtype"] = msgtype_tmp
except UnicodeDecodeError:
msg_dict["msgtype"] = "UNREADABLE"
msg_dict["body"] = msg_ser.read().hex()
msg_dict["error"] = "Unrecognized message type."
messages.append(msg_dict)
print(f"WARNING - Unrecognized message type {msgtype} in {path}", file=sys.stderr)
continue
# Deserialize the message
msg = MESSAGEMAP[msgtype]()
msg_dict["msgtype"] = msgtype.decode()
try:
msg.deserialize(msg_ser)
except KeyboardInterrupt:
raise
except Exception:
# Unable to deserialize message body
msg_ser.seek(0, os.SEEK_SET)
msg_dict["body"] = msg_ser.read().hex()
msg_dict["error"] = "Unable to deserialize message."
messages.append(msg_dict)
print(f"WARNING - Unable to deserialize message in {path}", file=sys.stderr)
continue
# Convert body of message into a jsonable object
if length:
msg_dict["body"] = to_jsonable(msg)
messages.append(msg_dict)
if progress_bar:
# Update the progress bar to the end of the current file
# in case we exited the loop early
f_in.seek(0, os.SEEK_END) # Go to end of file
diff = f_in.tell() - bytes_read - 1
progress_bar.update(diff)
def main():
parser = argparse.ArgumentParser(
description=__doc__,
epilog="EXAMPLE \n\t{0} -o out.json <data-dir>/message_capture/**/*.dat".format(sys.argv[0]),
formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument(
"capturepaths",
nargs='+',
help="binary message capture files to parse.")
parser.add_argument(
"-o", "--output",
help="output file. If unset print to stdout")
parser.add_argument(
"-n", "--no-progress-bar",
action='store_true',
help="disable the progress bar. Automatically set if the output is not a terminal")
args = parser.parse_args()
capturepaths = [Path.cwd() / Path(capturepath) for capturepath in args.capturepaths]
output = Path.cwd() / Path(args.output) if args.output else False
use_progress_bar = (not args.no_progress_bar) and sys.stdout.isatty()
messages = [] # type: List[Any]
if use_progress_bar:
total_size = sum(capture.stat().st_size for capture in capturepaths)
progress_bar = ProgressBar(total_size)
else:
progress_bar = None
for capture in capturepaths:
process_file(str(capture), messages, "recv" in capture.stem, progress_bar)
messages.sort(key=lambda msg: msg['time'])
if use_progress_bar:
progress_bar.set_progress(1)
jsonrep = json.dumps(messages)
if output:
with open(str(output), 'w+', encoding="utf8") as f_out:
f_out.write(jsonrep)
else:
print(jsonrep)
if __name__ == "__main__":
main()

5
contrib/qos/README.md Normal file
View file

@ -0,0 +1,5 @@
### QoS (Quality of service) ###
This is a Linux bash script that will set up tc to limit the outgoing bandwidth for connections to the DigiByte network. It limits outbound TCP traffic with a source or destination port of 8333, but not if the destination IP is within a LAN.
This means one can have an always-on digibyted instance running, and another local digibyted/digibyte-qt instance which connects to this node and receives blocks from it.

61
contrib/qos/tc.sh Executable file
View file

@ -0,0 +1,61 @@
# Copyright (c) 2017 The Bitcoin Core developers
# Copyright (c) 2017-2019 The DigiByte Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
#network interface on which to limit traffic
IF="eth0"
#limit of the network interface in question
LINKCEIL="1gbit"
#limit outbound DigiByte protocol traffic to this rate
LIMIT="160kbit"
#defines the IPv4 address space for which you wish to disable rate limiting
LOCALNET_V4="192.168.0.0/16"
#defines the IPv6 address space for which you wish to disable rate limiting
LOCALNET_V6="fe80::/10"
#delete existing rules ('Error: Cannot delete qdisc with handle of zero.' means there weren't any.)
tc qdisc del dev ${IF} root
#add root class
tc qdisc add dev ${IF} root handle 1: htb default 10
#add parent class
tc class add dev ${IF} parent 1: classid 1:1 htb rate ${LINKCEIL} ceil ${LINKCEIL}
#add our two classes. one unlimited, another limited
tc class add dev ${IF} parent 1:1 classid 1:10 htb rate ${LINKCEIL} ceil ${LINKCEIL} prio 0
tc class add dev ${IF} parent 1:1 classid 1:11 htb rate ${LIMIT} ceil ${LIMIT} prio 1
#add handles to our classes so packets marked with <x> go into the class with "... handle <x> fw ..."
tc filter add dev ${IF} parent 1: protocol ip prio 1 handle 1 fw classid 1:10
tc filter add dev ${IF} parent 1: protocol ip prio 2 handle 2 fw classid 1:11
if [ -n "${LOCALNET_V6}" ] ; then
# v6 cannot have the same priority value as v4
tc filter add dev ${IF} parent 1: protocol ipv6 prio 3 handle 1 fw classid 1:10
tc filter add dev ${IF} parent 1: protocol ipv6 prio 4 handle 2 fw classid 1:11
fi
#delete any existing rules
#disable for now
#ret=0
#while [ $ret -eq 0 ]; do
# iptables -t mangle -D OUTPUT 1
# ret=$?
#done
#limit outgoing traffic to and from port 8333. but not when dealing with a host on the local network
# (defined by $LOCALNET_V4 and $LOCALNET_V6)
# --set-mark marks packages matching these criteria with the number "2" (v4)
# --set-mark marks packages matching these criteria with the number "4" (v6)
# these packets are filtered by the tc filter with "handle 2"
# this filter sends the packages into the 1:11 class, and this class is limited to ${LIMIT}
iptables -t mangle -A OUTPUT -p tcp -m tcp --dport 8333 ! -d ${LOCALNET_V4} -j MARK --set-mark 0x2
iptables -t mangle -A OUTPUT -p tcp -m tcp --sport 8333 ! -d ${LOCALNET_V4} -j MARK --set-mark 0x2
if [ -n "${LOCALNET_V6}" ] ; then
ip6tables -t mangle -A OUTPUT -p tcp -m tcp --dport 8333 ! -d ${LOCALNET_V6} -j MARK --set-mark 0x4
ip6tables -t mangle -A OUTPUT -p tcp -m tcp --sport 8333 ! -d ${LOCALNET_V6} -j MARK --set-mark 0x4
fi

185
contrib/rpm/README.md Normal file
View file

@ -0,0 +1,185 @@
RPM Spec File Notes
-------------------
The RPM spec file provided here is for DigiByte-Core 0.12.0 and builds on CentOS
7 with either the CentOS provided OpenSSL library or with LibreSSL as packaged
at [LibreLAMP.com](https://librelamp.com/). It should hopefully not be too
difficult to port the RPM spec file to most RPM based Linux distributions.
When porting the spec file to build for a particular distribution, there are
some important notes.
## Sources
It is considered good form for all sources to reference a URL where the source
can be downloaded.
Sources 0-9 should be reserved for source code tarballs. `Source0` should
reference the release tarball available from https://digibyte.org/bin/ and
`Source1` should reference the BerkeleyDB source.
Sources 10-99 are for source files that are maintained in the
[DigiByte git repository](https://github.com/digibyte/digibyte) but are not part of
the release tarball. Most of these will reside in the `contrib` sub-directory.
Sources 10-19 should be reserved for miscellaneous configuration files.
Currently only `Source10` is used, for the example `digibyte.conf` file.
Sources 20-29 should be reserved for man pages. Currently only `Source20`
through `Source23` are used.
Sources 30-39 should be reserved for SELinux related files. Currently only
`Source30` through `Source32` are used. Until those files are in a tagged
release, the full URL specified in the RPM spec file will not work. You can get
them from the git repository where you retrieved this file.
Sources 100+ are for files that are not source tarballs and are not maintained
in the digibyte git repository. At present only an SVG version of the DigiByte
icon is used.
## Patches
In general, patches should be avoided. When a packager feels a patch is
necessary, the packager should bring the problem to the attention of the digibyte
developers so that an official fix to the issue can make it into the next
release.
### Patch0 digibyte-0.12.0-libressl.patch
This patch is only needed if building against LibreSSL. LibreSSL is not the
standard TLS library on most Linux distributions. The patch will likely not be
needed when 0.12.1 is released, a proper fix is already in the DigiByte git
master branch.
## BuildRequires
The packages specified in the `BuildRequires` are specified according to the
package naming convention currently used in CentOS 7 and EPEL for CentOS 7. You
may need to change some of the package names for other distributions. This is
most likely to be the case with the Qt packages.
## BerkeleyDB
The `build-unix.md` file recommends building against BerkeleyDB 4.8.30. Even if
that is the version your Linux distribution ships with, it probably is a good
idea to build DigiByte Core against a static version of that library compiled
according to the instructions in the `build-unix.md` file so that any changes
the distribution may make in the future will not result in a problem for users.
The problem that can exist, clients built against different versions of
BerkeleyDB may not be able read each other's `wallet.dat` file which can make it
difficult for a user to recover from backup in the event of a system failure.
## Graphical User Interface and Qt Version
The RPM spec file will by default build the GUI client linked against the Qt5
libraries. If you wish instead to link against the Qt4 libraries you need to
pass the switch `-D '_use_qt4 1'` at build time to the `rpmbuild` or `mock`
command used to build the packages.
If you would prefer not to build the GUI at all, you can pass the switch
`-D '_no_gui 1'` to the `rpmbuild` or `mock` build command.
## Desktop and KDE Files
The desktop and KDE meta files are created in the spec file itself with the
`cat` command. This is done to allow easy distribution specific changes without
needing to use any patches. A specific timestamp is given to the files so that
it does not they do not appear to have been updated every time the package is
built. If you do make changes to them, you probably should update timestamp
assigned to them in the `touch` command that specifies the timestamp.
## SVG, PNG, and XPM Icons
The `digibyte.svg` file is from the source listed as `Source100`. It is used as
the source for the PNG and XPM files. The generated PNG and XPM files are given
the same timestamp as the source SVG file as a means of indicating they are
derived from it.
## Systemd
This spec file assumes the target distribution uses systemd. That really only
matters for the `digibyte-server` package. At this point, most RPM based
distributions that still receive vendor updates do in fact use systemd.
The files to control the service are created in the RPM spec file itself using
the `cat` command. This is done to make it easy to modify for other
distributions that may implement things differently without needing to patch
source. A specific timestamp is given to the files so that they do not appear
to have been updated every time the package is built. If you do make changes to
them, you probably should update the timestamp assigned to them in the `touch`
command that specifies the timestamp.
## SELinux
The `digibyte-server` package should have SELinux support. How to properly do
that *may* vary by distribution and version of distribution.
The SELinux stuff in this RPM spec file *should* be correct for CentOS, RHEL,
and Fedora but it would be a good idea to review it before building the package
on other distributions.
## Tests
The `%check` section takes a very long time to run. If your build system has a
time limit for package build, you may need to make an exception for this
package. On CentOS 7 the `%check` section completes successfully with both
OpenSSL and LibreSSL, a failure really does mean something is wrong.
## LibreSSL Build Notes
To build against LibreSSL you will need to pass the switch
`-D '_use_libressl 1'` to the `rpmbuild` or `mock` command or the spec file will
want the OpenSSL development files.
### LibreSSL and Boost
LibreSSL (and some newer builds of OpenSSL) do not have support for SSLv3. This
can cause issues with the Boost package if the Boost package has not been
patched accordingly. On those distributions, you will either need to build
DigiByte-Core against OpenSSL or use a patched version of Boost in the build
system.
As SSLv3 is no longer safe, distributions that have not patched Boost to work
with TLS libraries that do not support SSLv3 should have bug reports filed
against the Boost package. This bug report has already been filed for RHEL 7 but
it may need to be filed for other distributions.
A patch for Boost: https://github.com/boostorg/asio/pull/23/files
## ZeroMQ
At this time, this RPM spec file does not support the ZeroMQ build options. A
suitable version of ZeroMQ is not available for the platform this spec file was
developed on (CentOS 7).
## Legacy Credit
This RPM spec file is largely based upon the work of Michael Hampton at
[Ringing Liberty](https://www.ringingliberty.com/digibyte/). He has been
packaging DigiByte for Fedora at least since 2012.
Most of the differences between his packaging and this package are stylistic in
nature. The major differences:
1. He builds from a github tagged release rather than a release tarball. This
should not result in different source code.
2. He does not build BerkeleyDB but instead uses the BerkeleyDB provided by the
Linux distribution. For the distributions he packages for, they currently all
use the same version of BerkeleyDB so that difference is *probably* just
academic.
3. As of his 10.11.2 package he did not allow for building against LibreSSL,
specifying a build without the Qt GUI, or specifying which version of the Qt
libraries to use.
4. I renamed the `digibyte` package that contains the Qt GUI to `digibyte-core` as
that appears to be how the general population refers to it, in contrast to
`digibyte-xt` or `digibyte-classic`. I wanted to make sure the general population
knows what they are getting when installing the GUI package.
As far as minor differences, I generally prefer to assign the file permissions
in the `%files` portion of an RPM spec file rather than specifying the
permissions of a file during `%install` and other minor things like that
are largely just cosmetic.

6
contrib/rpm/build.sh Normal file
View file

@ -0,0 +1,6 @@
#!/bin/bash
yum -y install epel-release; yum -y install git vim rpm-build rpmdevtools
mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
# git clone repo
spectool -g -R contrib/rpm/digibyte.spec
yum-builddep contrib/rpm/digibyte.spec

8
contrib/rpm/digibyte.fc Normal file
View file

@ -0,0 +1,8 @@
/usr/bin/digibyte-cli -- gen_context(system_u:object_r:digibyte_exec_t,s0)
/usr/sbin/digibyted -- gen_context(system_u:object_r:digibyte_exec_t,s0)
/usr/lib(64)?/digibyte/digibyted -- gen_context(system_u:object_r:digibyte_exec_t,s0)
/etc/digibyte(/.*)? gen_context(system_u:object_r:digibyte_conf_t,s0)
/var/lib/digibyte(/.*)? gen_context(system_u:object_r:digibyte_var_lib_t,s0)
(/var)?/run/digibyted(/.*)? gen_context(system_u:object_r:digibyte_var_run_t,s0)

157
contrib/rpm/digibyte.if Normal file
View file

@ -0,0 +1,157 @@
## <summary>policy for digibyte</summary>
########################################
## <summary>
## Transition to digibyte.
## </summary>
## <param name="domain">
## <summary>
## Domain allowed to transition.
## </summary>
## </param>
#
interface(`digibyte_domtrans',`
gen_require(`
type digibyte_t, digibyte_exec_t;
')
corecmd_search_bin($1)
domtrans_pattern($1, digibyte_exec_t, digibyte_t)
')
########################################
## <summary>
## Execute digibyte server in the digibyte domain.
## </summary>
## <param name="domain">
## <summary>
## Domain allowed access.
## </summary>
## </param>
#
interface(`digibyte_initrc_domtrans',`
gen_require(`
type digibyte_initrc_exec_t;
')
init_labeled_script_domtrans($1, digibyte_initrc_exec_t)
')
########################################
## <summary>
## Search digibyte lib directories.
## </summary>
## <param name="domain">
## <summary>
## Domain allowed access.
## </summary>
## </param>
#
interface(`digibyte_search_lib',`
gen_require(`
type digibyte_var_lib_t;
')
allow $1 digibyte_var_lib_t:dir search_dir_perms;
files_search_var_lib($1)
')
########################################
## <summary>
## Read digibyte lib files.
## </summary>
## <param name="domain">
## <summary>
## Domain allowed access.
## </summary>
## </param>
#
interface(`digibyte_read_lib_files',`
gen_require(`
type digibyte_var_lib_t;
')
files_search_var_lib($1)
read_files_pattern($1, digibyte_var_lib_t, digibyte_var_lib_t)
')
########################################
## <summary>
## Manage digibyte lib files.
## </summary>
## <param name="domain">
## <summary>
## Domain allowed access.
## </summary>
## </param>
#
interface(`digibyte_manage_lib_files',`
gen_require(`
type digibyte_var_lib_t;
')
files_search_var_lib($1)
manage_files_pattern($1, digibyte_var_lib_t, digibyte_var_lib_t)
')
########################################
## <summary>
## Manage digibyte lib directories.
## </summary>
## <param name="domain">
## <summary>
## Domain allowed access.
## </summary>
## </param>
#
interface(`digibyte_manage_lib_dirs',`
gen_require(`
type digibyte_var_lib_t;
')
files_search_var_lib($1)
manage_dirs_pattern($1, digibyte_var_lib_t, digibyte_var_lib_t)
')
########################################
## <summary>
## All of the rules required to administrate
## a digibyte environment
## </summary>
## <param name="domain">
## <summary>
## Domain allowed access.
## </summary>
## </param>
## <param name="role">
## <summary>
## Role allowed access.
## </summary>
## </param>
## <rolecap/>
#
interface(`digibyte_admin',`
gen_require(`
type digibyte_t;
type digibyte_initrc_exec_t;
type digibyte_var_lib_t;
')
allow $1 digibyte_t:process { ptrace signal_perms };
ps_process_pattern($1, digibyte_t)
digibyte_initrc_domtrans($1)
domain_system_change_exemption($1)
role_transition $2 digibyte_initrc_exec_t system_r;
allow $2 system_r;
files_search_var_lib($1)
admin_pattern($1, digibyte_var_lib_t)
')

439
contrib/rpm/digibyte.spec Normal file
View file

@ -0,0 +1,439 @@
%define bdbv 4.8.30
%global selinux_variants mls strict targeted
%if 0%{?_no_gui:1}
%define _buildqt 0
%define buildargs --with-gui=no
%else
%define _buildqt 1
%if 0%{?_use_qt4}
%define buildargs --with-qrencode --with-gui=qt4
%else
%define buildargs --with-qrencode --with-gui=qt5
%endif
%endif
Name: digibyte
Version: 7.17.2
Release: 1%{?dist}
Summary: Peer to Peer Cryptographic Currency
Group: Applications/System
License: MIT
URL: https://digibyte.org/
Source0: https://github.com/digibyte/digibyte/releases/download/v%{version}/digibyte-%{version}.tar.gz
Source1: http://download.oracle.com/berkeley-db/db-%{bdbv}.NC.tar.gz
Source10: https://raw.githubusercontent.com/digibyte/digibyte/v%{version}/contrib/debian/examples/digibyte.conf
#man pages
Source20: https://raw.githubusercontent.com/digibyte/digibyte/v%{version}/doc/man/digibyted.1
Source21: https://raw.githubusercontent.com/digibyte/digibyte/v%{version}/doc/man/digibyte-cli.1
Source22: https://raw.githubusercontent.com/digibyte/digibyte/v%{version}/doc/man/digibyte-qt.1
#selinux
Source30: https://raw.githubusercontent.com/digibyte/digibyte/v%{version}/contrib/rpm/digibyte.te
# Source31 - what about digibyte-tx and bench_digibyte ???
Source31: https://raw.githubusercontent.com/digibyte/digibyte/v%{version}/contrib/rpm/digibyte.fc
Source32: https://raw.githubusercontent.com/digibyte/digibyte/v%{version}/contrib/rpm/digibyte.if
Source100: http://icons.iconarchive.com/icons/cjdowner/cryptocurrency-flat/1024/DigiByte-DGB-icon.png
BuildRequires: openssl-devel
BuildRequires: boost-devel
BuildRequires: miniupnpc-devel
BuildRequires: autoconf automake libtool
BuildRequires: libevent-devel
BuildRequires: devtoolset-7-gcc
BuildRequires: devtoolset-7-gcc-c++
BuildRequires: ImageMagick
%description
DigiByte is a digital cryptographic currency that uses peer-to-peer technology to
operate with no central authority or banks; managing transactions and the
issuing of digibytes is carried out collectively by the network.
%if %{_buildqt}
%package core
Summary: Peer to Peer Cryptographic Currency
Group: Applications/System
Obsoletes: %{name} < %{version}-%{release}
Provides: %{name} = %{version}-%{release}
%if 0%{?_use_qt4}
BuildRequires: qt-devel
%else
BuildRequires: qt5-qtbase-devel
# for /usr/bin/lrelease-qt5
BuildRequires: qt5-linguist
%endif
BuildRequires: protobuf-devel
BuildRequires: qrencode-devel
BuildRequires: %{_bindir}/desktop-file-validate
# for icon generation from SVG
BuildRequires: %{_bindir}/inkscape
BuildRequires: %{_bindir}/convert
%description core
DigiByte is a digital cryptographic currency that uses peer-to-peer technology to
operate with no central authority or banks; managing transactions and the
issuing of digibytes is carried out collectively by the network.
This package contains the Qt based graphical client and node. If you are looking
to run a DigiByte wallet, this is probably the package you want.
%endif
%package libs
Summary: DigiByte shared libraries
Group: System Environment/Libraries
%description libs
This package provides the digibyteconsensus shared libraries. These libraries
may be used by third party software to provide consensus verification
functionality.
Unless you know need this package, you probably do not.
%package devel
Summary: Development files for digibyte
Group: Development/Libraries
Requires: %{name}-libs = %{version}-%{release}
%description devel
This package contains the header files and static library for the
digibyteconsensus shared library. If you are developing or compiling software
that wants to link against that library, then you need this package installed.
Most people do not need this package installed.
%package server
Summary: The digibyte daemon
Group: System Environment/Daemons
Requires: digibyte-utils = %{version}-%{release}
Requires: selinux-policy policycoreutils-python
Requires(pre): shadow-utils
Requires(post): %{_sbindir}/semodule %{_sbindir}/restorecon %{_sbindir}/fixfiles %{_sbindir}/sestatus
Requires(postun): %{_sbindir}/semodule %{_sbindir}/restorecon %{_sbindir}/fixfiles %{_sbindir}/sestatus
BuildRequires: systemd
BuildRequires: checkpolicy
BuildRequires: %{_datadir}/selinux/devel/Makefile
%description server
This package provides a stand-alone digibyte-core daemon. For most users, this
package is only needed if they need a full-node without the graphical client.
Some third party wallet software will want this package to provide the actual
digibyte-core node they use to connect to the network.
If you use the graphical digibyte-core client then you almost certainly do not
need this package.
%package utils
Summary: DigiByte utilities
Group: Applications/System
%description utils
This package provides several command line utilities for interacting with a
digibyte-core daemon.
The digibyte-cli utility allows you to communicate and control a digibyte daemon
over RPC, the digibyte-tx utility allows you to create a custom transaction, and
the bench_digibyte utility can be used to perform some benchmarks.
This package contains utilities needed by the digibyte-server package.
%prep
%setup -q
cp -p %{SOURCE10} ./digibyte.conf.example
tar -zxf %{SOURCE1}
cp -p db-%{bdbv}.NC/LICENSE ./db-%{bdbv}.NC-LICENSE
mkdir db4 SELinux
cp -p %{SOURCE30} %{SOURCE31} %{SOURCE32} SELinux/
%build
. /opt/rh/devtoolset-7/enable
CWD=`pwd`
cd db-%{bdbv}.NC/build_unix/
../dist/configure --enable-cxx --disable-shared --with-pic --prefix=${CWD}/db4
make install
cd ../..
./autogen.sh
%configure LDFLAGS="-L${CWD}/db4/lib/" CPPFLAGS="-I${CWD}/db4/include/" --with-miniupnpc --enable-glibc-back-compat %{buildargs}
make %{?_smp_mflags}
pushd SELinux
for selinuxvariant in %{selinux_variants}; do
make NAME=${selinuxvariant} -f %{_datadir}/selinux/devel/Makefile
mv digibyte.pp digibyte.pp.${selinuxvariant}
make NAME=${selinuxvariant} -f %{_datadir}/selinux/devel/Makefile clean
done
popd
%install
make install DESTDIR=%{buildroot}
mkdir -p -m755 %{buildroot}%{_sbindir}
mv %{buildroot}%{_bindir}/digibyted %{buildroot}%{_sbindir}/digibyted
# systemd stuff
mkdir -p %{buildroot}%{_tmpfilesdir}
cat <<EOF > %{buildroot}%{_tmpfilesdir}/digibyte.conf
d /run/digibyted 0750 digibyte digibyte -
EOF
touch -a -m -t 201504280000 %{buildroot}%{_tmpfilesdir}/digibyte.conf
mkdir -p %{buildroot}%{_sysconfdir}/sysconfig
cat <<EOF > %{buildroot}%{_sysconfdir}/sysconfig/digibyte
# Provide options to the digibyte daemon here, for example
# OPTIONS="-testnet -disable-wallet"
OPTIONS=""
# System service defaults.
# Don't change these unless you know what you're doing.
CONFIG_FILE="%{_sysconfdir}/digibyte/digibyte.conf"
DATA_DIR="%{_localstatedir}/lib/digibyte"
PID_FILE="/run/digibyted/digibyted.pid"
EOF
touch -a -m -t 201504280000 %{buildroot}%{_sysconfdir}/sysconfig/digibyte
mkdir -p %{buildroot}%{_unitdir}
cat <<EOF > %{buildroot}%{_unitdir}/digibyte.service
[Unit]
Description=DigiByte daemon
After=syslog.target network.target
[Service]
Type=forking
ExecStart=%{_sbindir}/digibyted -daemon -conf=\${CONFIG_FILE} -datadir=\${DATA_DIR} -pid=\${PID_FILE} \$OPTIONS
EnvironmentFile=%{_sysconfdir}/sysconfig/digibyte
User=digibyte
Group=digibyte
Restart=on-failure
PrivateTmp=true
TimeoutStopSec=120
TimeoutStartSec=60
StartLimitInterval=240
StartLimitBurst=5
[Install]
WantedBy=multi-user.target
EOF
touch -a -m -t 201504280000 %{buildroot}%{_unitdir}/digibyte.service
#end systemd stuff
mkdir %{buildroot}%{_sysconfdir}/digibyte
mkdir -p %{buildroot}%{_localstatedir}/lib/digibyte
#SELinux
for selinuxvariant in %{selinux_variants}; do
install -d %{buildroot}%{_datadir}/selinux/${selinuxvariant}
install -p -m 644 SELinux/digibyte.pp.${selinuxvariant} %{buildroot}%{_datadir}/selinux/${selinuxvariant}/digibyte.pp
done
%if %{_buildqt}
# qt icons
install -D -p share/pixmaps/digibyte.ico %{buildroot}%{_datadir}/pixmaps/digibyte.ico
install -p share/pixmaps/nsis-header.bmp %{buildroot}%{_datadir}/pixmaps/
install -p share/pixmaps/nsis-wizard.bmp %{buildroot}%{_datadir}/pixmaps/
install -p %{SOURCE100} %{buildroot}%{_datadir}/pixmaps/digibyte.svg
%{_bindir}/convert -resize 16x16 %{SOURCE100} %{buildroot}%{_datadir}/pixmaps/digibyte16.png
%{_bindir}/convert -resize 32x32 %{SOURCE100} %{buildroot}%{_datadir}/pixmaps/digibyte32.png
%{_bindir}/convert -resize 64x64 %{SOURCE100} %{buildroot}%{_datadir}/pixmaps/digibyte64.png
%{_bindir}/convert -resize 128x128 %{SOURCE100} %{buildroot}%{_datadir}/pixmaps/digibyte128.png
%{_bindir}/convert -resize 256x256 %{SOURCE100} %{buildroot}%{_datadir}/pixmaps/digibyte256.png
%{_bindir}/convert -resize 16x16 %{buildroot}%{_datadir}/pixmaps/digibyte256.png %{buildroot}%{_datadir}/pixmaps/digibyte16.xpm
%{_bindir}/convert -resize 32x32 %{buildroot}%{_datadir}/pixmaps/digibyte256.png %{buildroot}%{_datadir}/pixmaps/digibyte32.xpm
%{_bindir}/convert -resize 64x64 %{buildroot}%{_datadir}/pixmaps/digibyte256.png %{buildroot}%{_datadir}/pixmaps/digibyte64.xpm
%{_bindir}/convert -resize 128x128 %{buildroot}%{_datadir}/pixmaps/digibyte256.png %{buildroot}%{_datadir}/pixmaps/digibyte128.xpm
%{_bindir}/convert %{buildroot}%{_datadir}/pixmaps/digibyte256.png %{buildroot}%{_datadir}/pixmaps/digibyte256.xpm
touch %{buildroot}%{_datadir}/pixmaps/*.png -r %{SOURCE100}
touch %{buildroot}%{_datadir}/pixmaps/*.xpm -r %{SOURCE100}
# Desktop File - change the touch timestamp if modifying
mkdir -p %{buildroot}%{_datadir}/applications
cat <<EOF > %{buildroot}%{_datadir}/applications/digibyte-core.desktop
[Desktop Entry]
Encoding=UTF-8
Name=DigiByte
Comment=DigiByte P2P Cryptocurrency
Comment[fr]=DigiByte, monnaie virtuelle cryptographique pair à pair
Comment[tr]=DigiByte, eşten eşe kriptografik sanal para birimi
Exec=digibyte-qt %u
Terminal=false
Type=Application
Icon=digibyte128
MimeType=x-scheme-handler/digibyte;
Categories=Office;Finance;
EOF
# change touch date when modifying desktop
touch -a -m -t 201511100546 %{buildroot}%{_datadir}/applications/digibyte-core.desktop
%{_bindir}/desktop-file-validate %{buildroot}%{_datadir}/applications/digibyte-core.desktop
# KDE protocol - change the touch timestamp if modifying
mkdir -p %{buildroot}%{_datadir}/kde4/services
cat <<EOF > %{buildroot}%{_datadir}/kde4/services/digibyte-core.protocol
[Protocol]
exec=digibyte-qt '%u'
protocol=digibyte
input=none
output=none
helper=true
listing=
reading=false
writing=false
makedir=false
deleting=false
EOF
# change touch date when modifying protocol
touch -a -m -t 201511100546 %{buildroot}%{_datadir}/kde4/services/digibyte-core.protocol
%endif
# man pages
install -D -p %{SOURCE20} %{buildroot}%{_mandir}/man1/digibyted.1
install -p %{SOURCE21} %{buildroot}%{_mandir}/man1/digibyte-cli.1
%if %{_buildqt}
install -p %{SOURCE22} %{buildroot}%{_mandir}/man1/digibyte-qt.1
%endif
# nuke these, we do extensive testing of binaries in %%check before packaging
rm -f %{buildroot}%{_bindir}/test_*
# %check
# make check
# srcdir=src test/digibyte-util-test.py
# test/functional/test_runner.py --extended
%post libs -p /sbin/ldconfig
%postun libs -p /sbin/ldconfig
%pre server
getent group digibyte >/dev/null || groupadd -r digibyte
getent passwd digibyte >/dev/null ||
useradd -r -g digibyte -d /var/lib/digibyte -s /sbin/nologin \
-c "DigiByte wallet server" digibyte
exit 0
%post server
%systemd_post digibyte.service
# SELinux
if [ `%{_sbindir}/sestatus |grep -c "disabled"` -eq 0 ]; then
for selinuxvariant in %{selinux_variants}; do
%{_sbindir}/semodule -s ${selinuxvariant} -i %{_datadir}/selinux/${selinuxvariant}/digibyte.pp &> /dev/null || :
done
%{_sbindir}/semanage port -a -t digibyte_port_t -p tcp 8332
%{_sbindir}/semanage port -a -t digibyte_port_t -p tcp 8333
%{_sbindir}/semanage port -a -t digibyte_port_t -p tcp 18332
%{_sbindir}/semanage port -a -t digibyte_port_t -p tcp 18333
%{_sbindir}/semanage port -a -t digibyte_port_t -p tcp 18443
%{_sbindir}/semanage port -a -t digibyte_port_t -p tcp 18444
%{_sbindir}/fixfiles -R digibyte-server restore &> /dev/null || :
%{_sbindir}/restorecon -R %{_localstatedir}/lib/digibyte || :
fi
%posttrans server
%{_bindir}/systemd-tmpfiles --create
%preun server
%systemd_preun digibyte.service
%postun server
%systemd_postun digibyte.service
# SELinux
if [ $1 -eq 0 ]; then
if [ `%{_sbindir}/sestatus |grep -c "disabled"` -eq 0 ]; then
%{_sbindir}/semanage port -d -p tcp 8332
%{_sbindir}/semanage port -d -p tcp 8333
%{_sbindir}/semanage port -d -p tcp 18332
%{_sbindir}/semanage port -d -p tcp 18333
%{_sbindir}/semanage port -d -p tcp 18443
%{_sbindir}/semanage port -d -p tcp 18444
for selinuxvariant in %{selinux_variants}; do
%{_sbindir}/semodule -s ${selinuxvariant} -r digibyte &> /dev/null || :
done
%{_sbindir}/fixfiles -R digibyte-server restore &> /dev/null || :
[ -d %{_localstatedir}/lib/digibyte ] && \
%{_sbindir}/restorecon -R %{_localstatedir}/lib/digibyte &> /dev/null || :
fi
fi
%clean
rm -rf %{buildroot}
%if %{_buildqt}
%files core
%defattr(-,root,root,-)
%license COPYING db-%{bdbv}.NC-LICENSE
%doc COPYING digibyte.conf.example doc/README.md doc/bips.md doc/files.md doc/reduce-traffic.md doc/release-notes.md doc/tor.md
%attr(0755,root,root) %{_bindir}/digibyte-qt
%attr(0644,root,root) %{_datadir}/applications/digibyte-core.desktop
%attr(0644,root,root) %{_datadir}/kde4/services/digibyte-core.protocol
%attr(0644,root,root) %{_datadir}/pixmaps/*.ico
%attr(0644,root,root) %{_datadir}/pixmaps/*.bmp
%attr(0644,root,root) %{_datadir}/pixmaps/*.svg
%attr(0644,root,root) %{_datadir}/pixmaps/*.png
%attr(0644,root,root) %{_datadir}/pixmaps/*.xpm
%attr(0644,root,root) %{_mandir}/man1/digibyte-qt.1*
%endif
%files libs
%defattr(-,root,root,-)
%license COPYING
%doc COPYING doc/README.md doc/shared-libraries.md
%{_libdir}/lib*.so.*
%files devel
%defattr(-,root,root,-)
%license COPYING
%doc COPYING doc/README.md doc/developer-notes.md doc/shared-libraries.md
%attr(0644,root,root) %{_includedir}/*.h
%{_libdir}/*.so
%{_libdir}/*.a
%{_libdir}/*.la
%attr(0644,root,root) %{_libdir}/pkgconfig/*.pc
%files server
%defattr(-,root,root,-)
%license COPYING db-%{bdbv}.NC-LICENSE
%doc COPYING digibyte.conf.example doc/README.md doc/REST-interface.md doc/bips.md doc/dnsseed-policy.md doc/files.md doc/reduce-traffic.md doc/release-notes.md doc/tor.md
%attr(0755,root,root) %{_sbindir}/digibyted
%attr(0644,root,root) %{_tmpfilesdir}/digibyte.conf
%attr(0644,root,root) %{_unitdir}/digibyte.service
%dir %attr(0750,digibyte,digibyte) %{_sysconfdir}/digibyte
%dir %attr(0750,digibyte,digibyte) %{_localstatedir}/lib/digibyte
%config(noreplace) %attr(0600,root,root) %{_sysconfdir}/sysconfig/digibyte
%attr(0644,root,root) %{_datadir}/selinux/*/*.pp
%attr(0644,root,root) %{_mandir}/man1/digibyted.1*
%files utils
%defattr(-,root,root,-)
%license COPYING
%doc COPYING digibyte.conf.example doc/README.md
%attr(0755,root,root) %{_bindir}/digibyte-cli
%attr(0755,root,root) %{_bindir}/digibyte-tx
%attr(0755,root,root) %{_bindir}/bench_digibyte
%attr(0644,root,root) %{_mandir}/man1/digibyte-cli.1*
%attr(0644,root,root) %{_mandir}/man1/digibyte-tx.1*
%changelog
* Fri Feb 26 2016 Alice Wonder <buildmaster@librelamp.com> - 0.12.0-2
- Rename Qt package from digibyte to digibyte-core
- Make building of the Qt package optional
- When building the Qt package, default to Qt5 but allow building
- against Qt4
- Only run SELinux stuff in post scripts if it is not set to disabled
* Wed Feb 24 2016 Alice Wonder <buildmaster@librelamp.com> - 0.12.0-1
- Initial spec file for 0.12.0 release
# This spec file is written from scratch but a lot of the packaging decisions are directly
# based upon the 0.11.2 package spec file from https://www.ringingliberty.com/digibyte/

81
contrib/rpm/digibyte.te Normal file
View file

@ -0,0 +1,81 @@
policy_module(digibyte, 1.100.1)
########################################
#
# Declarations
#
type digibyte_t;
type digibyte_exec_t;
init_daemon_domain(digibyte_t, digibyte_exec_t)
permissive digibyte_t;
type digibyte_initrc_exec_t;
init_script_file(digibyte_initrc_exec_t)
type digibyte_conf_t;
files_type(digibyte_conf_t)
type digibyte_var_lib_t;
files_type(digibyte_var_lib_t)
type digibyte_var_run_t;
files_type(digibyte_var_run_t)
type digibyte_port_t;
corenet_port(digibyte_port_t)
########################################
#
# digibyte local policy
#
allow digibyte_t self:process { fork };
allow digibyte_t self:fifo_file rw_fifo_file_perms;
allow digibyte_t self:unix_stream_socket create_stream_socket_perms;
manage_dirs_pattern(digibyte_t, digibyte_conf_t, digibyte_conf_t)
manage_files_pattern(digibyte_t, digibyte_conf_t, digibyte_conf_t)
manage_dirs_pattern(digibyte_t, digibyte_var_lib_t, digibyte_var_lib_t)
manage_files_pattern(digibyte_t, digibyte_var_lib_t, digibyte_var_lib_t)
files_var_lib_filetrans(digibyte_t, digibyte_var_lib_t, { dir file })
manage_dirs_pattern(digibyte_t, digibyte_var_run_t, digibyte_var_run_t)
manage_files_pattern(digibyte_t, digibyte_var_run_t, digibyte_var_run_t)
sysnet_dns_name_resolve(digibyte_t)
corenet_all_recvfrom_unlabeled(digibyte_t)
allow digibyte_t self:tcp_socket create_stream_socket_perms;
corenet_tcp_sendrecv_generic_if(digibyte_t)
corenet_tcp_sendrecv_generic_node(digibyte_t)
corenet_tcp_sendrecv_all_ports(digibyte_t)
corenet_tcp_bind_generic_node(digibyte_t)
gen_require(`
type digibyte_port_t;
')
allow digibyte_t digibyte_port_t:tcp_socket name_bind;
gen_require(`
type digibyte_port_t;
')
allow digibyte_t digibyte_port_t:tcp_socket name_connect;
domain_use_interactive_fds(digibyte_t)
files_read_etc_files(digibyte_t)
miscfiles_read_localization(digibyte_t)
sysnet_dns_name_resolve(digibyte_t)
allow digibyte_t digibyte_exec_t:file execute_no_trans;
allow digibyte_t self:process setsched;
corecmd_exec_ls(digibyte_t)
corenet_tcp_connect_http_port(digibyte_t)
dev_read_urand(digibyte_t)
fs_getattr_xattr_fs(digibyte_t)
kernel_read_system_state(digibyte_t)

1
contrib/seeds/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
seeds_main.txt

27
contrib/seeds/README.md Normal file
View file

@ -0,0 +1,27 @@
# Seeds
Utility to generate the seeds.txt list that is compiled into the client
(see [src/chainparamsseeds.h](/src/chainparamsseeds.h) and other utilities in [contrib/seeds](/contrib/seeds)).
Be sure to update `PATTERN_AGENT` in `makeseeds.py` to include the current version,
and remove old versions as necessary (at a minimum when GetDesirableServiceFlags
changes its default return value, as those are the services which seeds are added
to addrman with).
The seeds compiled into the release are created from sipa's DNS seed data, like this:
curl -s http://digibyte.sipa.be/seeds.txt.gz | gzip -dc > seeds_main.txt
python3 makeseeds.py < seeds_main.txt > nodes_main.txt
python3 generate-seeds.py . > ../../src/chainparamsseeds.h
## Dependencies
Ubuntu, Debian:
sudo apt-get install python3-dnspython
and/or for other operating systems:
pip install dnspython
See https://dnspython.readthedocs.io/en/latest/installation.html for more information.

Some files were not shown because too many files have changed in this diff Show more