6 ===============================================================================
7 crypto -- Encryption Layer for the Deltatar Backup
8 ===============================================================================
12 - AES-GCM for the symmetric encryption;
17 - NIST Recommendation for Block Cipher Modes of Operation: Galois/Counter
19 http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf
22 https://cryptome.org/2014/01/aes-gcm-v1.pdf
24 - Authentication weaknesses in GCM
25 http://csrc.nist.gov/groups/ST/toolkit/BCM/documents/comments/CWC-GCM/Ferguson2.pdf
27 Trouble with python-cryptography packages: authentication tags can only be
28 passed in advance: https://github.com/pyca/cryptography/pull/3421
31 -------------------------------------------------------------------------------
33 Errors fall into roughly three categories:
35 - Cryptographical errors or invalid data.
37 - ``InvalidGCMTag`` (decryption failed on account of an invalid GCM
39 - ``InvalidIVFixedPart`` (IV fixed part of object not found in list),
40 - ``DuplicateIV`` (the IV of an encrypted object already occurred),
41 - ``DecryptionError`` (used in CLI decryption for presenting error
42 conditions to the user).
44 - Incorrect usage of the library.
46 - ``InvalidParameter`` (non-conforming user supplied parameter),
47 - ``InvalidHeader`` (data passed for reading not parsable into header),
48 - ``FormatError`` (cannot handle header or parameter version),
51 - Bad internal state. If one of these is encountered it means that a state
52 was reached that shouldn’t occur during normal processing.
57 Also, ``EndOfFile`` is used as a sentinel to communicate that a stream supplied
58 for reading is exhausted.
60 Initialization Vectors
61 -------------------------------------------------------------------------------
63 Initialization vectors are checked reuse during the lifetime of a decryptor.
64 The fixed counters for metadata files cannot be reused and attempts to do so
65 will cause a DuplicateIV error. This means the length of objects encrypted with
66 a metadata counter is capped at 63 GB.
68 For ordinary, non-metadata payload, there is an optional mode with strict IV
69 checking that causes a crypto context to fail if an IV encountered or created
70 was already used for decrypting or encrypting, respectively, an earlier object.
71 Note that this mode can trigger false positives when decrypting non-linearly,
72 e. g. when traversing the same object multiple times. Since the crypto context
73 has no notion of a position in a PDT encrypted archive, this condition must be
74 sorted out downstream.
77 -------------------------------------------------------------------------------
79 ``crypto.py`` may be invoked as a script for decrypting, validating, and
80 splitting PDT encrypted files. Consult the usage message for details.
84 Decrypt from stdin using the password ‘foo’: ::
86 $ crypto.py process foo -i - -o - <some-file.tar.gz.pdtcrypt >some-file.tar.gz
88 Output verbose information about the encrypted objects in the archive: ::
90 $ crypto.py process foo -v -i some-file.tar.gz.pdtcrypt -o /dev/null
91 PDT: decrypt from some-file.tar.gz.pdtcrypt
92 PDT: decrypt to /dev/null
93 PDT: source: file some-file.tar.gz.pdtcrypt
94 PDT: sink: file /dev/null
96 PDT: · version = 1 : 0100
97 PDT: · paramversion = 1 : 0100
98 PDT: · nacl : d270 b031 00d1 87e2 c946 610d 7b7f 7e5f
99 PDT: · iv : 02ee 3dd7 a963 1eb1 0100 0000
100 PDT: · ctsize = 591 : 4f02 0000 0000 0000
101 PDT: · tag : 5b2d 6d8b 8f82 4842 12fd 0b10 b6e3 369b
102 PDT: 64 decrypt obj no. 1, 591 B
103 PDT: · [64] 0% done, read block (591 B of 591 B remaining)
104 PDT: · decrypt ciphertext 591 B
105 PDT: · decrypt plaintext 591 B
109 Also, the mode *scrypt* allows deriving encryption keys. To calculate the
110 encryption key from the password ‘foo’ and the salt of the first object in a
111 PDT encrypted file: ::
113 $ crypto.py scrypt foo -i some-file.pdtcrypt
114 {"paramversion": 1, "salt": "Cqzbk48e3peEjzWto8D0yA==", "key": "JH9EkMwaM4x9F5aim5gK/Q=="}
116 The computed 16 byte key is given in hexadecimal notation in the value to
117 ``hash`` and can be fed into Python’s ``binascii.unhexlify()`` to obtain the
118 corresponding binary representation.
120 Note that in Scrypt hashing mode, no data integrity checks are being performed.
121 If the wrong password is given, a wrong key will be derived. Whether the password
122 was indeed correct can only be determined by decrypting. Note that since PDT
123 archives essentially consist of a stream of independent objects, the salt and
124 other parameters may change. Thus a key derived using above method from the
125 first object doesn’t necessarily apply to any of the subsequent objects.
134 from functools import reduce, partial
142 except ImportError as exn:
145 if __name__ == "__main__": ## Work around the import mechanism’s lest Python’s
146 pwd = os.getcwd() ## preference for local imports causes a cyclical
147 ## import (crypto → pylibscrypt → […] → ./tarfile → crypto).
148 sys.path = [ p for p in sys.path if p.find ("deltatar") < 0 ]
151 from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
152 from cryptography.hazmat.backends import default_backend
156 __all__ = [ "hdr_make", "hdr_read", "hdr_fmt", "hdr_fmt_pretty"
158 , "PDTCRYPT_HDR_SIZE", "AES_GCM_IV_CNT_DATA"
159 , "AES_GCM_IV_CNT_INFOFILE", "AES_GCM_IV_CNT_INDEX"
163 ###############################################################################
165 ###############################################################################
167 class EndOfFile (Exception):
171 def __init__ (self, n=None, msg=None):
177 class InvalidParameter (Exception):
178 """Inputs not valid for PDT encryption."""
182 class InvalidHeader (Exception):
183 """Header not valid."""
187 class InvalidGCMTag (Exception):
189 The GCM tag calculated during decryption differs from that in the object
195 class InvalidIVFixedPart (Exception):
197 IV fixed part not in supplied list: either the backup is corrupt or the
198 current object does not belong to it.
203 class IVFixedPartError (Exception):
205 Error creating a unique IV fixed part: repeated calls to system RNG yielded
206 the same sequence of bytes as the last IV used.
211 class InvalidFileCounter (Exception):
213 When encrypting, an attempted reuse of a dedicated counter (info file,
214 index file) was caught.
219 class DuplicateIV (Exception):
221 During encryption, the current IV fixed part is identical to an already
222 existing IV (same prefix and file counter). This indicates tampering or
223 programmer error and cannot be recovered from.
228 class NonConsecutiveIV (Exception):
230 IVs not numbered consecutively. This is a hard error with strict IV
231 checking. Precludes random access to the encrypted objects.
236 class FormatError (Exception):
237 """Unusable parameters in header."""
241 class DecryptionError (Exception):
242 """Error during decryption with ``crypto.py`` on the command line."""
246 class Unreachable (Exception):
248 Makeshift __builtin_unreachable(); always a programmer error if
254 class InternalError (Exception):
255 """Errors not ascribable to bad user inputs or cryptography."""
259 ###############################################################################
260 ## crypto layer version
261 ###############################################################################
263 ENCRYPTION_PARAMETERS = \
265 { "kdf": ("dummy", 16)
266 , "enc": "passthrough" }
274 , "enc": "aes-gcm" } }
276 ###############################################################################
278 ###############################################################################
280 PDTCRYPT_HDR_MAGIC = b"PDTCRYPT"
282 PDTCRYPT_HDR_SIZE_MAGIC = 8 # 8
283 PDTCRYPT_HDR_SIZE_VERSION = 2 # 10
284 PDTCRYPT_HDR_SIZE_PARAMVERSION = 2 # 12
285 PDTCRYPT_HDR_SIZE_NACL = 16 # 28
286 PDTCRYPT_HDR_SIZE_IV = 12 # 40
287 PDTCRYPT_HDR_SIZE_CTSIZE = 8 # 48
288 PDTCRYPT_HDR_SIZE_TAG = 16 # 64 GCM auth tag
290 PDTCRYPT_HDR_SIZE = PDTCRYPT_HDR_SIZE_MAGIC + PDTCRYPT_HDR_SIZE_VERSION \
291 + PDTCRYPT_HDR_SIZE_PARAMVERSION + PDTCRYPT_HDR_SIZE_NACL \
292 + PDTCRYPT_HDR_SIZE_IV + PDTCRYPT_HDR_SIZE_CTSIZE \
293 + PDTCRYPT_HDR_SIZE_TAG # = 64
295 # precalculate offsets since Python can’t do constant folding over names
296 HDR_OFF_VERSION = PDTCRYPT_HDR_SIZE_MAGIC
297 HDR_OFF_PARAMVERSION = HDR_OFF_VERSION + PDTCRYPT_HDR_SIZE_VERSION
298 HDR_OFF_NACL = HDR_OFF_PARAMVERSION + PDTCRYPT_HDR_SIZE_PARAMVERSION
299 HDR_OFF_IV = HDR_OFF_NACL + PDTCRYPT_HDR_SIZE_NACL
300 HDR_OFF_CTSIZE = HDR_OFF_IV + PDTCRYPT_HDR_SIZE_IV
301 HDR_OFF_TAG = HDR_OFF_CTSIZE + PDTCRYPT_HDR_SIZE_CTSIZE
305 FMT_I2N_IV = "<8sL" # 8 random bytes ‖ 32 bit counter
306 FMT_I2N_HDR = ("<" # host byte order
310 "16s" # sodium chloride
316 AES_GCM_MAX_SIZE = (1 << 36) - (1 << 5) # 2^39 - 2^8 b ≅ 64 GB
317 PDTCRYPT_MAX_OBJ_SIZE_DEFAULT = 63 * (1 << 30) # 63 GB
318 PDTCRYPT_MAX_OBJ_SIZE = PDTCRYPT_MAX_OBJ_SIZE_DEFAULT
320 # index and info files are written on-the fly while encrypting so their
321 # counters must be available inadvance
322 AES_GCM_IV_CNT_INFOFILE = 1 # constant
323 AES_GCM_IV_CNT_INDEX = AES_GCM_IV_CNT_INFOFILE + 1
324 AES_GCM_IV_CNT_DATA = AES_GCM_IV_CNT_INDEX + 1 # also for multivolume
325 AES_GCM_IV_CNT_MAX_DEFAULT = 0xffFFffFF
326 AES_GCM_IV_CNT_MAX = AES_GCM_IV_CNT_MAX_DEFAULT
328 # IV structure and generation
329 PDTCRYPT_IV_GEN_MAX_RETRIES = 10 # ×
330 PDTCRYPT_IV_FIXEDPART_SIZE = 8 # B
331 PDTCRYPT_IV_COUNTER_SIZE = 4 # B
333 ###############################################################################
335 ###############################################################################
341 # , paramversion : u16
347 # fn hdr_read (f : handle) -> hdrinfo;
348 # fn hdr_make (f : handle, h : hdrinfo) -> IOResult<usize>;
349 # fn hdr_fmt (h : hdrinfo) -> String;
354 Read bytes as header structure.
356 If the input could not be interpreted as a header, fail with
361 mag, version, paramversion, nacl, iv, ctsize, tag = \
362 struct.unpack (FMT_I2N_HDR, data)
363 except Exception as exn:
364 raise InvalidHeader ("error unpacking header from [%r]: %s"
365 % (binascii.hexlify (data), str (exn)))
367 if mag != PDTCRYPT_HDR_MAGIC:
368 raise InvalidHeader ("bad magic in header: expected [%s], got [%s]"
369 % (PDTCRYPT_HDR_MAGIC, mag))
372 { "version" : version
373 , "paramversion" : paramversion
381 def hdr_read_stream (instr):
383 Read header from stream at the current position.
385 Fail with ``InvalidHeader`` if insufficient bytes were read from the
386 stream, or if the content could not be interpreted as a header.
388 data = instr.read(PDTCRYPT_HDR_SIZE)
392 elif ldata != PDTCRYPT_HDR_SIZE:
393 raise InvalidHeader ("hdr_read_stream: expected %d B, received %d B"
394 % (PDTCRYPT_HDR_SIZE, ldata))
395 return hdr_read (data)
398 def hdr_from_params (version, paramversion, nacl, iv, ctsize, tag):
400 Assemble the necessary values into a PDTCRYPT header.
402 :type version: int to fit uint16_t
403 :type paramversion: int to fit uint16_t
404 :type nacl: bytes to fit uint8_t[16]
405 :type iv: bytes to fit uint8_t[12]
406 :type size: int to fit uint64_t
407 :type tag: bytes to fit uint8_t[16]
409 buf = bytearray (PDTCRYPT_HDR_SIZE)
410 bufv = memoryview (buf)
413 struct.pack_into (FMT_I2N_HDR, bufv, 0,
415 version, paramversion, nacl, iv, ctsize, tag)
416 except Exception as exn:
417 return False, "error assembling header: %s" % str (exn)
419 return True, bytes (buf)
422 def hdr_make_dummy (s):
424 Create a header sized block of bytes initialized to a value derived from a
425 string. Used to verify we’ve jumped back correctly to the actual position
426 of the object header.
428 c = reduce (lambda a, c: a + ord(c), s, 0) % 0xFF
429 return bytes (bytearray (struct.pack ("B", c)) * PDTCRYPT_HDR_SIZE)
434 Assemble a header from the given header structure.
436 return hdr_from_params (version=hdr.get("version"),
437 paramversion=hdr.get("paramversion"),
438 nacl=hdr.get("nacl"), iv=hdr.get("iv"),
439 ctsize=hdr.get("ctsize"), tag=hdr.get("tag"))
442 HDR_FMT = "I2n_header { version: %d, paramversion: %d, nacl: %s[%d]," \
443 " iv: %s[%d], ctsize: %d, tag: %s[%d] }"
446 """Format a header structure into readable output."""
447 return HDR_FMT % (h["version"], h["paramversion"],
448 binascii.hexlify (h["nacl"]), len(h["nacl"]),
449 binascii.hexlify (h["iv"]), len(h["iv"]),
451 binascii.hexlify (h["tag"]), len(h["tag"]))
454 def hex_spaced_of_bytes (b):
455 """Format bytes object, hexdump style."""
456 return " ".join ([ "%.2x%.2x" % (c1, c2)
457 for c1, c2 in zip (b[0::2], b[1::2]) ]) \
458 + (len (b) | 1 == len (b) and " %.2x" % b[-1] or "") # odd lengths
461 def hdr_iv_counter (h):
462 """Extract the variable part of the IV of the given header."""
463 _fixed, cnt = struct.unpack (FMT_I2N_IV, h ["iv"])
467 def hdr_iv_fixed (h):
468 """Extract the fixed part of the IV of the given header."""
469 fixed, _cnt = struct.unpack (FMT_I2N_IV, h ["iv"])
473 hdr_dump = hex_spaced_of_bytes
477 """version = %-4d : %s
478 paramversion = %-4d : %s
485 def hdr_fmt_pretty (h):
487 Format header structure into multi-line representation of its contents and
488 their raw representation. (Omit the implicit “PDTCRYPT” magic bytes that
489 precede every header.)
491 return HDR_FMT_PRETTY \
493 hex_spaced_of_bytes (struct.pack (FMT_UINT16_LE, h["version"])),
495 hex_spaced_of_bytes (struct.pack (FMT_UINT16_LE, h["paramversion"])),
496 hex_spaced_of_bytes (h["nacl"]),
497 hex_spaced_of_bytes (h["iv"]),
499 hex_spaced_of_bytes (struct.pack (FMT_UINT64_LE, h["ctsize"])),
500 hex_spaced_of_bytes (h["tag"]))
502 IV_FMT = "((f %s) (c %d))"
505 """Format the two components of an IV in a readable fashion."""
506 fixed, cnt = struct.unpack (FMT_I2N_IV, iv)
507 return IV_FMT % (binascii.hexlify (fixed), cnt)
510 ###############################################################################
511 ## passthrough / null encryption
512 ###############################################################################
514 class PassthroughCipher (object):
516 tag = struct.pack ("<QQ", 0, 0)
518 def __init__ (self) : pass
520 def update (self, b) : return b
522 def finalize (self) : return b""
524 def finalize_with_tag (self, _) : return b""
526 ###############################################################################
527 ## convenience wrapper
528 ###############################################################################
531 def kdf_dummy (klen, password, _nacl):
533 Fake KDF for testing purposes that is called when parameter version zero is
536 q, r = divmod (klen, len (password))
537 if isinstance (password, bytes) is False:
538 password = password.encode ()
539 return password * q + password [:r], b""
542 SCRYPT_KEY_MEMO = { } # static because needed for both the info file and the archive
545 def kdf_scrypt (params, password, nacl):
547 Wrapper for the Scrypt KDF, corresponds to parameter version one. The
548 computation result is memoized based on the inputs to facilitate spawning
549 multiple encryption contexts.
554 dkLen = params["dkLen"]
557 nacl = os.urandom (params["NaCl_LEN"])
559 key_parms = (password, nacl, N, r, p, dkLen)
560 global SCRYPT_KEY_MEMO
561 if key_parms not in SCRYPT_KEY_MEMO:
562 SCRYPT_KEY_MEMO [key_parms] = \
563 pylibscrypt.scrypt (password, nacl, N, r, p, dkLen)
564 return SCRYPT_KEY_MEMO [key_parms], nacl
567 def kdf_by_version (paramversion=None, defs=None):
569 Pick the KDF handler corresponding to the parameter version or the
572 :rtype: function (password : str, nacl : str) -> str
574 if paramversion is not None:
575 defs = ENCRYPTION_PARAMETERS.get(paramversion, None)
577 raise InvalidParameter ("no encryption parameters for version %r"
579 (kdf, params) = defs["kdf"]
581 if kdf == "scrypt" : fn = kdf_scrypt
582 if kdf == "dummy" : fn = kdf_dummy
584 raise ValueError ("key derivation method %r unknown" % kdf)
585 return partial (fn, params)
588 ###############################################################################
590 ###############################################################################
592 def scrypt_hashsource (pw, ins):
594 Calculate the SCRYPT hash from the password and the information contained
595 in the first header found in ``ins``.
597 This does not validate whether the first object is encrypted correctly.
599 if isinstance (pw, str) is True:
601 elif isinstance (pw, bytes) is False:
602 raise InvalidParameter ("password must be a string, not %s"
604 if isinstance (ins, io.BufferedReader) is False and \
605 isinstance (ins, io.FileIO) is False:
606 raise InvalidParameter ("file to hash must be opened in “binary” mode")
609 hdr = hdr_read_stream (ins)
610 except EndOfFile as exn:
611 noise ("PDT: malformed input: end of file reading first object header")
616 pver = hdr ["paramversion"]
617 if PDTCRYPT_VERBOSE is True:
618 noise ("PDT: salt of first object : %s" % binascii.hexlify (nacl))
619 noise ("PDT: parameter version of archive : %d" % pver)
622 defs = ENCRYPTION_PARAMETERS.get(pver, None)
623 kdfname, params = defs ["kdf"]
624 if kdfname != "scrypt":
625 noise ("PDT: input is not an SCRYPT archive")
628 kdf = kdf_by_version (None, defs)
629 except ValueError as exn:
630 noise ("PDT: object has unknown parameter version %d" % pver)
632 hsh, _void = kdf (pw, nacl)
634 return hsh, nacl, hdr ["version"], pver
637 def scrypt_hashfile (pw, fname):
639 Calculate the SCRYPT hash from the password and the information contained
640 in the first header found in the given file. The header is read only at
643 with deptdcrypt_mk_stream (PDTCRYPT_SOURCE, fname or "-") as ins:
644 hsh, _void, _void, _void = scrypt_hashsource (pw, ins)
648 ###############################################################################
650 ###############################################################################
652 class Crypto (object):
654 Encryption context to remain alive throughout an entire tarfile pass.
659 cnt = None # file counter (uint32_t != 0)
660 iv = None # current IV
661 fixed = None # accu for 64 bit fixed parts of IV
662 used_ivs = None # tracks IVs
663 strict_ivs = False # if True, panic on duplicate object IV
672 info_counter_used = False
673 index_counter_used = False
675 def __init__ (self, *al, **akv):
676 self.used_ivs = set ()
677 self.set_parameters (*al, **akv)
680 def next_fixed (self):
685 def set_object_counter (self, cnt=None):
687 Safely set the internal counter of encrypted objects. Numerous
690 The same counter may not be reused in combination with one IV fixed
691 part. This is validated elsewhere in the IV handling.
693 Counter zero is invalid. The first two counters are reserved for
694 metadata. The implementation does not allow for splitting metadata
695 files over multiple encrypted objects. (This would be possible by
696 assigning new fixed parts.) Thus in a Deltatar backup there is at most
697 one object with a counter value of one and two. On creation of a
698 context, the initial counter may be chosen. The globals
699 ``AES_GCM_IV_CNT_INFOFILE`` and ``AES_GCM_IV_CNT_INDEX`` can be used to
700 request one of the reserved values. If one of these values has been
701 used, any further attempt of setting the counter to that value will
702 be rejected with an ``InvalidFileCounter`` exception.
704 Out of bounds values (i. e. below one and more than the maximum of 2³²)
705 cause an ``InvalidParameter`` exception to be thrown.
708 self.cnt = AES_GCM_IV_CNT_DATA
710 if cnt == 0 or cnt > AES_GCM_IV_CNT_MAX + 1:
711 raise InvalidParameter ("invalid counter value %d requested: "
712 "acceptable values are from 1 to %d"
713 % (cnt, AES_GCM_IV_CNT_MAX))
714 if cnt == AES_GCM_IV_CNT_INFOFILE:
715 if self.info_counter_used is True:
716 raise InvalidFileCounter ("attempted to reuse info file "
717 "counter %d: must be unique" % cnt)
718 self.info_counter_used = True
719 elif cnt == AES_GCM_IV_CNT_INDEX:
720 if self.index_counter_used is True:
721 raise InvalidFileCounter ("attempted to reuse index file "
722 " counter %d: must be unique" % cnt)
723 self.index_counter_used = True
724 if cnt <= AES_GCM_IV_CNT_MAX:
727 # cnt == AES_GCM_IV_CNT_MAX + 1 → wrap
728 self.cnt = AES_GCM_IV_CNT_DATA
732 def set_parameters (self, password=None, key=None, paramversion=None,
733 nacl=None, counter=None, strict_ivs=False):
735 Configure the internal state of a crypto context. Not intended for
739 self.set_object_counter (counter)
740 self.strict_ivs = strict_ivs
742 if paramversion is not None:
743 self.paramversion = paramversion
746 self.key, self.nacl = key, nacl
749 if password is not None:
750 if isinstance (password, bytes) is False:
751 password = str.encode (password)
752 self.password = password
753 if paramversion is None and nacl is None:
754 # postpone key setup until first header is available
756 kdf = kdf_by_version (paramversion)
758 self.key, self.nacl = kdf (password, nacl)
761 def process (self, buf):
763 Encrypt / decrypt a buffer. Invokes the ``.update()`` method on the
764 wrapped encryptor or decryptor, respectively.
766 The Cryptography exception ``AlreadyFinalized`` is translated to an
767 ``InternalError`` at this point. It may occur in sound code when the GC
768 closes an encrypting stream after an error. Everywhere else it must be
772 raise RuntimeError ("process: context not initialized")
773 self.stats ["in"] += len (buf)
775 out = self.enc.update (buf)
776 except cryptography.exceptions.AlreadyFinalized as exn:
777 raise InternalError (exn)
778 self.stats ["out"] += len (out)
782 def next (self, password, paramversion, nacl, iv):
784 Prepare for encrypting another object: Reset the data counters and
785 change the configuration in case one of the variable parameters differs
786 from the last object. Also check the IV for duplicates and error out
787 if strict checking was requested.
791 self.stats ["obj"] += 1
793 self.check_duplicate_iv (iv)
795 if ( self.paramversion != paramversion
796 or self.password != password
797 or self.nacl != nacl):
798 self.set_parameters (password=password, paramversion=paramversion,
799 nacl=nacl, strict_ivs=self.strict_ivs)
802 def check_duplicate_iv (self, iv):
804 Add an IV (the 12 byte representation as in the header) to the list. With
805 strict checking enabled, this will throw a ``DuplicateIV``. Depending on
806 the context, this may indicate a serious error (IV reuse).
808 if self.strict_ivs is True and iv in self.used_ivs:
809 raise DuplicateIV ("iv %s was reused" % iv_fmt (iv))
810 # vi has not been used before; add to collection
811 self.used_ivs.add (iv)
816 Access the data counters.
818 return self.stats ["obj"], self.stats ["in"], self.stats ["out"]
821 class Encrypt (Crypto):
827 def __init__ (self, version, paramversion, password=None, key=None, nacl=None,
828 counter=AES_GCM_IV_CNT_DATA, strict_ivs=True):
830 The ctor will throw immediately if one of the parameters does not conform
833 counter=AES_GCM_IV_CNT_DATA, strict_ivs=True):
834 :type version: int to fit uint16_t
835 :type paramversion: int to fit uint16_t
836 :param password: mutually exclusive with ``key``
837 :type password: bytes
838 :param key: mutually exclusive with ``password``
841 :type counter: initial object counter the values
842 ``AES_GCM_IV_CNT_INFOFILE`` and
843 ``AES_GCM_IV_CNT_INDEX`` are unique in each backup set
844 and cannot be reused even with different fixed parts.
845 :type strict_ivs: bool
847 if password is None and key is None \
848 or password is not None and key is not None :
849 raise InvalidParameter ("__init__: need either key or password")
852 if isinstance (key, bytes) is False:
853 raise InvalidParameter ("__init__: key must be provided as "
854 "bytes, not %s" % type (key))
856 raise InvalidParameter ("__init__: salt must be provided along "
857 "with encryption key")
858 else: # password, no key
859 if isinstance (password, str) is False:
860 raise InvalidParameter ("__init__: password must be a string, not %s"
862 if len (password) == 0:
863 raise InvalidParameter ("__init__: supplied empty password but not "
864 "permitted for PDT encrypted files")
866 if isinstance (version, int) is False:
867 raise InvalidParameter ("__init__: version number must be an "
868 "integer, not %s" % type (version))
870 raise InvalidParameter ("__init__: version number must be a "
871 "nonnegative integer, not %d" % version)
873 if isinstance (paramversion, int) is False:
874 raise InvalidParameter ("__init__: crypto parameter version number "
875 "must be an integer, not %s"
876 % type (paramversion))
878 raise InvalidParameter ("__init__: crypto parameter version number "
879 "must be a nonnegative integer, not %d"
883 if isinstance (nacl, bytes) is False:
884 raise InvalidParameter ("__init__: salt given, but of type %s "
885 "instead of bytes" % type (nacl))
886 # salt length would depend on the actual encryption so it can’t be
887 # validated at this point
889 self.version = version
890 self.paramenc = ENCRYPTION_PARAMETERS.get (paramversion) ["enc"]
892 super().__init__ (password, key, paramversion, nacl, counter=counter,
893 strict_ivs=strict_ivs)
896 def next_fixed (self, retries=PDTCRYPT_IV_GEN_MAX_RETRIES):
898 Generate the next IV fixed part by reading eight bytes from
899 ``/dev/urandom``. The buffer so obtained is tested against the fixed
900 parts used so far to prevent accidental reuse of IVs. After a
901 configurable number of attempts to create a unique fixed part, it will
902 refuse to continue with an ``IVFixedPartError``. This is unlikely to
903 ever happen on a normal system but may detect an issue with the random
906 The list of fixed parts that were used by the context at hand can be
907 accessed through the ``.fixed`` list. Its last element is the fixed
908 part currently in use.
912 fp = os.urandom (PDTCRYPT_IV_FIXEDPART_SIZE)
913 if fp not in self.fixed:
914 self.fixed.append (fp)
917 raise IVFixedPartError ("error obtaining a unique IV fixed part from "
918 "/dev/urandom; giving up after %d tries" % i)
923 Construct a 12-bytes IV from the current fixed part and the object
926 return struct.pack(FMT_I2N_IV, self.fixed [-1], self.cnt)
929 def next (self, filename=None, counter=None):
931 Prepare for encrypting the next incoming object. Update the counter
932 and put together the IV, possibly changing prefixes. Then create the
935 The argument ``counter`` can be used to specify a file counter for this
936 object. Unless it is one of the reserved values, the counter of
937 subsequent objects will be computed from this one.
939 If this is the first object in a series, ``filename`` is required,
940 otherwise it is reused if not present. The value is used to derive a
941 header sized placeholder to use until after encryption when all the
942 inputs to construct the final header are available. This is then
943 matched in ``.done()`` against the value found at the position of the
944 header. The motivation for this extra check is primarily to assist
945 format debugging: It makes stray headers easy to spot in malformed
949 if self.lastinfo is None:
950 raise InvalidParameter ("next: filename is mandatory for "
952 filename, _dummy = self.lastinfo
954 if isinstance (filename, str) is False:
955 raise InvalidParameter ("next: filename must be a string, no %s"
957 if counter is not None:
958 if isinstance (counter, int) is False:
959 raise InvalidParameter ("next: the supplied counter is of "
960 "invalid type %s; please pass an "
961 "integer instead" % type (counter))
962 self.set_object_counter (counter)
964 self.iv = self.iv_make ()
965 if self.paramenc == "aes-gcm":
967 ( algorithms.AES (self.key)
968 , modes.GCM (self.iv)
969 , backend = default_backend ()) \
971 elif self.paramenc == "passthrough":
972 self.enc = PassthroughCipher ()
974 raise InvalidParameter ("next: parameter version %d not known"
976 hdrdum = hdr_make_dummy (filename)
977 self.lastinfo = (filename, hdrdum)
978 super().next (self.password, self.paramversion, self.nacl, self.iv)
980 self.set_object_counter (self.cnt + 1)
984 def done (self, cmpdata):
986 Complete encryption of an object. After this has been called, attempts
987 of encrypting further data will cause an error until ``.next()`` is
990 Returns a 64 bytes buffer containing the object header including all
991 values including the “late” ones e. g. the ciphertext size and the
994 if isinstance (cmpdata, bytes) is False:
995 raise InvalidParameter ("done: comparison input expected as bytes, "
996 "not %s" % type (cmpdata))
997 if self.lastinfo is None:
998 raise RuntimeError ("done: encryption context not initialized")
999 filename, hdrdum = self.lastinfo
1000 if cmpdata != hdrdum:
1001 raise RuntimeError ("done: bad sync of header for object %d: "
1002 "preliminary data does not match; this likely "
1003 "indicates a wrongly repositioned stream"
1005 data = self.enc.finalize ()
1006 self.stats ["out"] += len (data)
1007 self.ctsize += len (data)
1008 ok, hdr = hdr_from_params (self.version, self.paramversion, self.nacl,
1009 self.iv, self.ctsize, self.enc.tag)
1011 raise InternalError ("error constructing header: %r" % hdr)
1012 return data, hdr, self.fixed
1015 def process (self, buf):
1017 Encrypt a chunk of plaintext with the active encryptor. Returns the
1018 size of the input consumed. This **must** be checked downstream. If the
1019 maximum possible object size has been reached, the current context must
1020 be finalized and a new one established before any further data can be
1021 encrypted. The second argument is the remainder of the plaintext that
1022 was not encrypted for the caller to use immediately after the new
1025 if isinstance (buf, bytes) is False:
1026 raise InvalidParameter ("process: expected byte buffer, not %s"
1029 newptsize = self.ptsize + bsize
1030 diff = newptsize - PDTCRYPT_MAX_OBJ_SIZE
1033 newptsize = PDTCRYPT_MAX_OBJ_SIZE
1034 self.ptsize = newptsize
1035 data = super().process (buf [:bsize])
1036 self.ctsize += len (data)
1040 class Decrypt (Crypto):
1042 tag = None # GCM tag, part of header
1043 last_iv = None # check consecutive ivs in strict mode
1045 def __init__ (self, password=None, key=None, counter=None, fixedparts=None,
1048 Sanitizing ctor for the decryption context. ``fixedparts`` specifies a
1049 list of IV fixed parts accepted during decryption. If a fixed part is
1050 encountered that is not in the list, decryption will fail.
1052 :param password: mutually exclusive with ``key``
1053 :type password: bytes
1054 :param key: mutually exclusive with ``password``
1056 :type counter: initial object counter the values
1057 ``AES_GCM_IV_CNT_INFOFILE`` and
1058 ``AES_GCM_IV_CNT_INDEX`` are unique in each backup set
1059 and cannot be reused even with different fixed parts.
1060 :type fixedparts: bytes list
1062 if password is None and key is None \
1063 or password is not None and key is not None :
1064 raise InvalidParameter ("__init__: need either key or password")
1067 if isinstance (key, bytes) is False:
1068 raise InvalidParameter ("__init__: key must be provided as "
1069 "bytes, not %s" % type (key))
1070 else: # password, no key
1071 if isinstance (password, str) is False:
1072 raise InvalidParameter ("__init__: password must be a string, not %s"
1074 if len (password) == 0:
1075 raise InvalidParameter ("__init__: supplied empty password but not "
1076 "permitted for PDT encrypted files")
1078 if fixedparts is not None:
1079 if isinstance (fixedparts, list) is False:
1080 raise InvalidParameter ("__init__: IV fixed parts must be "
1081 "supplied as list, not %s"
1082 % type (fixedparts))
1083 self.fixed = fixedparts
1086 super().__init__ (password=password, key=key, counter=counter,
1087 strict_ivs=strict_ivs)
1090 def valid_fixed_part (self, iv):
1092 Check if a fixed part was already seen.
1094 # check if fixed part is known
1095 fixed, _cnt = struct.unpack (FMT_I2N_IV, iv)
1096 i = bisect.bisect_left (self.fixed, fixed)
1097 return i != len (self.fixed) and self.fixed [i] == fixed
1100 def check_consecutive_iv (self, iv):
1102 Check whether the counter part of the given IV is indeed the successor
1103 of the currently present counter. This should always be the case for
1104 the objects in a well formed PDT archive but should not be enforced
1105 when decrypting out-of-order.
1107 fixed, cnt = struct.unpack (FMT_I2N_IV, iv)
1108 if self.strict_ivs is True \
1109 and self.last_iv is not None \
1110 and self.last_iv [0] == fixed \
1111 and self.last_iv [1] != cnt - 1:
1112 raise NonConsecutiveIV ("iv %s counter not successor of "
1113 "last object (expected %d, found %d)"
1114 % (iv_fmt (self.last_iv [1]), cnt))
1115 self.last_iv = (iv, cnt)
1118 def next (self, hdr):
1120 Start decrypting the next object. The PDTCRYPT header for the object
1121 can be given either as already parsed object or as bytes.
1123 if isinstance (hdr, bytes) is True:
1124 hdr = hdr_read (hdr)
1125 elif isinstance (hdr, dict) is False:
1126 # this won’t catch malformed specs though
1127 raise InvalidParameter ("next: wrong type of parameter hdr: "
1128 "expected bytes or spec, got %s"
1131 paramversion = hdr ["paramversion"]
1136 raise InvalidHeader ("next: not a header %r" % hdr)
1138 super().next (self.password, paramversion, nacl, iv)
1139 if self.fixed is not None and self.valid_fixed_part (iv) is False:
1140 raise InvalidIVFixedPart ("iv %s has invalid fixed part"
1142 self.check_consecutive_iv (iv)
1145 defs = ENCRYPTION_PARAMETERS.get (paramversion, None)
1147 raise FormatError ("header contains unknown parameter version %d; "
1148 "maybe the file was created by a more recent "
1149 "version of Deltatar" % paramversion)
1151 if enc == "aes-gcm":
1153 ( algorithms.AES (self.key)
1154 , modes.GCM (iv, tag=self.tag)
1155 , backend = default_backend ()) \
1157 elif enc == "passthrough":
1158 self.enc = PassthroughCipher ()
1160 raise InternalError ("encryption parameter set %d refers to unknown "
1161 "mode %r" % (paramversion, enc))
1162 self.set_object_counter (self.cnt + 1)
1165 def done (self, tag=None):
1167 Stop decryption of the current object and finalize it with the active
1168 context. This will throw an *InvalidGCMTag* exception to indicate that
1169 the authentication tag does not match the data. If the tag is correct,
1170 the rest of the plaintext is returned.
1175 data = self.enc.finalize ()
1177 if isinstance (tag, bytes) is False:
1178 raise InvalidParameter ("done: wrong type of parameter "
1179 "tag: expected bytes, got %s"
1181 data = self.enc.finalize_with_tag (self.tag)
1182 except cryptography.exceptions.InvalidTag:
1183 raise InvalidGCMTag ("done: tag mismatch of object %d: %s "
1184 "rejected by finalize ()"
1185 % (self.cnt, binascii.hexlify (self.tag)))
1186 self.ctsize += len (data)
1187 self.stats ["out"] += len (data)
1191 def process (self, buf):
1193 Decrypt the bytes object *buf* with the active decryptor.
1195 if isinstance (buf, bytes) is False:
1196 raise InvalidParameter ("process: expected byte buffer, not %s"
1198 self.ctsize += len (buf)
1199 data = super().process (buf)
1200 self.ptsize += len (data)
1204 ###############################################################################
1206 ###############################################################################
1208 def _patch_global (glob, vow, n=None):
1210 Adapt upper file counter bound for testing IV logic. Completely unsafe.
1212 assert vow == "I am fully aware that this will void my warranty."
1213 r = globals () [glob]
1215 n = globals () [glob + "_DEFAULT"]
1216 globals () [glob] = n
1219 _testing_set_AES_GCM_IV_CNT_MAX = \
1220 partial (_patch_global, "AES_GCM_IV_CNT_MAX")
1222 _testing_set_PDTCRYPT_MAX_OBJ_SIZE = \
1223 partial (_patch_global, "PDTCRYPT_MAX_OBJ_SIZE")
1225 ###############################################################################
1226 ## freestanding invocation
1227 ###############################################################################
1229 PDTCRYPT_SUB_PROCESS = 0
1230 PDTCRYPT_SUB_SCRYPT = 1
1233 { "process" : PDTCRYPT_SUB_PROCESS
1234 , "scrypt" : PDTCRYPT_SUB_SCRYPT }
1236 PDTCRYPT_SECRET_PW = 0
1237 PDTCRYPT_SECRET_KEY = 1
1239 PDTCRYPT_DECRYPT = 1 << 0 # decrypt archive with password
1240 PDTCRYPT_SPLIT = 1 << 1 # split archive into individual objects
1241 PDTCRYPT_HASH = 1 << 2 # output scrypt hash for file and given password
1243 PDTCRYPT_SPLITNAME = "pdtcrypt-object-%d.bin"
1245 PDTCRYPT_VERBOSE = False
1246 PDTCRYPT_STRICTIVS = False
1247 PDTCRYPT_OVERWRITE = False
1248 PDTCRYPT_BLOCKSIZE = 1 << 12
1253 PDTCRYPT_DEFAULT_VER = 1
1254 PDTCRYPT_DEFAULT_PVER = 1
1256 # scrypt hashing output control
1257 PDTCRYPT_SCRYPT_INTRANATOR = 0
1258 PDTCRYPT_SCRYPT_PARAMETERS = 1
1259 PDTCRYPT_SCRYPT_DEFAULT = PDTCRYPT_SCRYPT_INTRANATOR
1261 PDTCRYPT_SCRYPT_FORMAT = \
1262 { "i2n" : PDTCRYPT_SCRYPT_INTRANATOR
1263 , "params" : PDTCRYPT_SCRYPT_PARAMETERS }
1266 class PDTDecryptionError (Exception):
1267 """Decryption failed."""
1269 class PDTSplitError (Exception):
1270 """Decryption failed."""
1273 def noise (*a, **b):
1274 print (file=sys.stderr, *a, **b)
1277 class PassthroughDecryptor (object):
1279 curhdr = None # write current header on first data write
1281 def __init__ (self):
1282 if PDTCRYPT_VERBOSE is True:
1283 noise ("PDT: no encryption; data passthrough")
1285 def next (self, hdr):
1286 ok, curhdr = hdr_make (hdr)
1288 raise PDTDecryptionError ("bad header %r" % hdr)
1289 self.curhdr = curhdr
1292 if self.curhdr is not None:
1296 def process (self, d):
1297 if self.curhdr is not None:
1303 def depdtcrypt (mode, secret, ins, outs):
1305 Remove PDTCRYPT layer from all objects encrypted with the secret. Used on a
1306 Deltatar backup this will yield a (possibly Gzip compressed) tarball.
1308 ctleft = -1 # length of ciphertext to consume
1309 ctcurrent = 0 # total ciphertext of current object
1310 total_obj = 0 # total number of objects read
1311 total_pt = 0 # total plaintext bytes
1312 total_ct = 0 # total ciphertext bytes
1313 total_read = 0 # total bytes read
1314 outfile = None # Python file object for output
1316 if mode & PDTCRYPT_DECRYPT: # decryptor
1318 if ks == PDTCRYPT_SECRET_PW:
1319 decr = Decrypt (password=secret [1], strict_ivs=PDTCRYPT_STRICTIVS)
1320 elif ks == PDTCRYPT_SECRET_KEY:
1321 key = binascii.unhexlify (secret [1])
1322 decr = Decrypt (key=key, strict_ivs=PDTCRYPT_STRICTIVS)
1324 raise InternalError ("‘%d’ does not specify a valid kind of secret"
1327 decr = PassthroughDecryptor ()
1330 """Dummy for non-split mode: output file does not vary."""
1333 if mode & PDTCRYPT_SPLIT:
1334 def nextout (outfile):
1336 We were passed an fd as outs for accessing the destination
1337 directory where extracted archive components are supposed
1342 if PDTCRYPT_VERBOSE is True:
1343 noise ("PDT: no output file to close at this point")
1345 if PDTCRYPT_VERBOSE is True:
1346 noise ("PDT: release output file %r" % outfile)
1347 # cleanup happens automatically by the GC; the next
1348 # line will error out on account of an invalid fd
1351 assert total_obj > 0
1352 fname = PDTCRYPT_SPLITNAME % total_obj
1354 oflags = os.O_CREAT | os.O_WRONLY
1355 if PDTCRYPT_OVERWRITE is True:
1356 oflags |= os.O_TRUNC
1359 outfd = os.open (fname, oflags, 0o600, dir_fd=outs)
1360 if PDTCRYPT_VERBOSE is True:
1361 noise ("PDT: new output file %s → %d" % (fname, outfd))
1362 except FileExistsError as exn:
1363 noise ("PDT: refusing to overwrite existing file %s" % fname)
1365 raise PDTSplitError ("destination file %s already exists"
1368 return os.fdopen (outfd, "wb", closefd=True)
1372 """ESPIPE is normal on stdio stream."""
1375 except OSError as exn:
1379 def out (pt, outfile):
1383 if PDTCRYPT_VERBOSE is True:
1384 noise ("PDT:\t· decrypt plaintext %d B" % (npt))
1386 nn = outfile.write (pt)
1387 except OSError as exn: # probably ENOSPC
1388 raise DecryptionError ("error (%s)" % exn)
1390 raise DecryptionError ("write aborted after %d of %d B" % (nn, npt))
1394 # current object completed; in a valid archive this marks either
1395 # the start of a new header or the end of the input
1396 if ctleft == 0: # current object requires finalization
1397 if PDTCRYPT_VERBOSE is True:
1398 noise ("PDT: %d finalize" % tell (ins))
1401 except InvalidGCMTag as exn:
1402 raise DecryptionError ("error finalizing object %d (%d B): "
1403 "%r" % (total_obj, len (pt), exn)) \
1406 if PDTCRYPT_VERBOSE is True:
1407 noise ("PDT:\t· object validated")
1409 if PDTCRYPT_VERBOSE is True:
1410 noise ("PDT: %d hdr" % tell (ins))
1412 hdr = hdr_read_stream (ins)
1413 total_read += PDTCRYPT_HDR_SIZE
1414 except EndOfFile as exn:
1415 total_read += exn.remainder
1416 if total_ct + total_obj * PDTCRYPT_HDR_SIZE != total_read:
1417 raise PDTDecryptionError ("ciphertext processed (%d B) plus "
1418 "overhead (%d × %d B) does not match "
1419 "the number of bytes read (%d )"
1420 % (total_ct, total_obj, PDTCRYPT_HDR_SIZE,
1422 # the single good exit
1423 return total_read, total_obj, total_ct, total_pt
1424 except InvalidHeader as exn:
1425 raise PDTDecryptionError ("invalid header at position %d in %r "
1426 "(%s)" % (tell (ins), exn, ins))
1427 if PDTCRYPT_VERBOSE is True:
1428 pretty = hdr_fmt_pretty (hdr)
1429 noise (reduce (lambda a, e: (a + "\n" if a else "") + "PDT:\t· " + e,
1430 pretty.splitlines (), ""))
1431 ctcurrent = ctleft = hdr ["ctsize"]
1435 total_obj += 1 # used in file counter with split mode
1437 # finalization complete or skipped in case of first object in
1438 # stream; create a new output file if necessary
1439 outfile = nextout (outfile)
1441 if PDTCRYPT_VERBOSE is True:
1442 noise ("PDT: %d decrypt obj no. %d, %d B"
1443 % (tell (ins), total_obj, ctleft))
1445 # always allocate a new buffer since python-cryptography doesn’t allow
1446 # passing a bytearray :/
1447 nexpect = min (ctleft, PDTCRYPT_BLOCKSIZE)
1448 if PDTCRYPT_VERBOSE is True:
1449 noise ("PDT:\t· [%d] %d%% done, read block (%d B of %d B remaining)"
1451 100 - ctleft * 100 / (ctcurrent > 0 and ctcurrent or 1),
1453 ct = ins.read (nexpect)
1457 raise EndOfFile (nct,
1458 "hit EOF after %d of %d B in block [%d:%d); "
1459 "%d B ciphertext remaining for object no %d"
1460 % (nct, nexpect, off, off + nexpect, ctleft,
1466 if PDTCRYPT_VERBOSE is True:
1467 noise ("PDT:\t· decrypt ciphertext %d B" % (nct))
1468 pt = decr.process (ct)
1472 def deptdcrypt_mk_stream (kind, path):
1473 """Create stream from file or stdio descriptor."""
1474 if kind == PDTCRYPT_SINK:
1476 if PDTCRYPT_VERBOSE is True: noise ("PDT: sink: stdout")
1477 return sys.stdout.buffer
1479 if PDTCRYPT_VERBOSE is True: noise ("PDT: sink: file %s" % path)
1480 return io.FileIO (path, "w")
1481 if kind == PDTCRYPT_SOURCE:
1483 if PDTCRYPT_VERBOSE is True: noise ("PDT: source: stdin")
1484 return sys.stdin.buffer
1486 if PDTCRYPT_VERBOSE is True: noise ("PDT: source: file %s" % path)
1487 return io.FileIO (path, "r")
1489 raise ValueError ("bogus stream “%s” / %s" % (kind, path))
1492 def mode_depdtcrypt (mode, secret, ins, outs):
1494 total_read, total_obj, total_ct, total_pt = \
1495 depdtcrypt (mode, secret, ins, outs)
1496 except DecryptionError as exn:
1497 noise ("PDT: Decryption failed:")
1499 noise ("PDT: “%s”" % exn)
1501 noise ("PDT: Did you specify the correct key / password?")
1504 except PDTSplitError as exn:
1505 noise ("PDT: Split operation failed:")
1507 noise ("PDT: “%s”" % exn)
1509 noise ("PDT: Hint: target directory should be empty.")
1513 if PDTCRYPT_VERBOSE is True:
1514 noise ("PDT: decryption successful" )
1515 noise ("PDT: %.10d bytes read" % total_read)
1516 noise ("PDT: %.10d objects decrypted" % total_obj )
1517 noise ("PDT: %.10d bytes ciphertext" % total_ct )
1518 noise ("PDT: %.10d bytes plaintext" % total_pt )
1524 def mode_scrypt (pw, ins=None, nacl=None, fmt=PDTCRYPT_SCRYPT_INTRANATOR):
1526 paramversion = PDTCRYPT_DEFAULT_PVER
1528 hsh, nacl, version, paramversion = scrypt_hashsource (pw, ins)
1529 defs = ENCRYPTION_PARAMETERS.get(paramversion, None)
1531 nacl = binascii.unhexlify (nacl)
1532 defs = ENCRYPTION_PARAMETERS.get(paramversion, None)
1533 version = PDTCRYPT_DEFAULT_VER
1535 kdfname, params = defs ["kdf"]
1537 kdf = kdf_by_version (None, defs)
1538 hsh, _void = kdf (pw, nacl)
1542 if fmt == PDTCRYPT_SCRYPT_INTRANATOR:
1543 out = json.dumps ({ "salt" : base64.b64encode (nacl).decode ()
1544 , "key" : base64.b64encode (hsh) .decode ()
1545 , "paramversion" : paramversion })
1546 elif fmt == PDTCRYPT_SCRYPT_PARAMETERS:
1547 out = json.dumps ({ "salt" : binascii.hexlify (nacl).decode ()
1548 , "key" : binascii.hexlify (hsh) .decode ()
1549 , "version" : version
1550 , "scrypt_params" : { "N" : params ["N"]
1551 , "r" : params ["r"]
1552 , "p" : params ["p"]
1553 , "dkLen" : params ["dkLen"] } })
1555 raise RuntimeError ("bad scrypt output scheme %r" % fmt)
1561 def usage (err=False):
1565 indent = ' ' * len (SELF)
1566 out ("usage: %s SUBCOMMAND { --help" % SELF)
1567 out (" %s | [ -v ] { -p PASSWORD | -k KEY }" % indent)
1568 out (" %s [ { -i | --in } { - | SOURCE } ]" % indent)
1569 out (" %s [ { -n | --nacl } { SALT } ]" % indent)
1570 out (" %s [ { -o | --out } { - | DESTINATION } ]" % indent)
1571 out (" %s [ -D | --no-decrypt ] [ -S | --split ]" % indent)
1572 out (" %s [ -f | --format ]" % indent)
1575 out ("\t\tSUBCOMMAND main mode: { process | scrypt }")
1577 out ("\t\t process: extract objects from PDT archive")
1578 out ("\t\t scrypt: calculate hash from password and first object")
1579 out ("\t\t-p PASSWORD password to derive the encryption key from")
1580 out ("\t\t-k KEY encryption key as 16 bytes in hexadecimal notation")
1581 out ("\t\t-s enforce strict handling of initialization vectors")
1582 out ("\t\t-i SOURCE file name to read from")
1583 out ("\t\t-o DESTINATION file to write output to")
1584 out ("\t\t-n SALT provide salt for scrypt mode in hex encoding")
1585 out ("\t\t-v print extra info")
1586 out ("\t\t-S split into files at object boundaries; this")
1587 out ("\t\t requires DESTINATION to refer to directory")
1588 out ("\t\t-D PDT header and ciphertext passthrough")
1589 out ("\t\t-f format of SCRYPT hash output (“default” or “parameters”)")
1591 out ("\tinstead of filenames, “-” may used to specify stdin / stdout")
1593 sys.exit ((err is True) and 42 or 0)
1603 def parse_argv (argv):
1605 mode = PDTCRYPT_DECRYPT
1610 scrypt_format = PDTCRYPT_SCRYPT_DEFAULT
1613 SELF = os.path.basename (next (argvi))
1616 rawsubcmd = next (argvi)
1617 subcommand = PDTCRYPT_SUB [rawsubcmd]
1618 except StopIteration:
1619 bail ("ERROR: subcommand required")
1621 bail ("ERROR: invalid subcommand “%s” specified" % rawsubcmd)
1627 except StopIteration:
1628 bail ("ERROR: argument list incomplete")
1630 def checked_secret (t, arg):
1635 bail ("ERROR: encountered “%s” but secret already given" % arg)
1638 if arg in [ "-h", "--help" ]:
1641 elif arg in [ "-v", "--verbose", "--wtf" ]:
1642 global PDTCRYPT_VERBOSE
1643 PDTCRYPT_VERBOSE = True
1644 elif arg in [ "-i", "--in", "--source" ]:
1645 insspec = checked_arg ()
1646 if PDTCRYPT_VERBOSE is True: noise ("PDT: decrypt from %s" % insspec)
1647 elif arg in [ "-p", "--password" ]:
1648 arg = checked_arg ()
1649 checked_secret (PDTCRYPT_SECRET_PW, arg)
1650 if PDTCRYPT_VERBOSE is True: noise ("PDT: decrypting with password")
1652 if subcommand == PDTCRYPT_SUB_PROCESS:
1653 if arg in [ "-s", "--strict-ivs" ]:
1654 global PDTCRYPT_STRICTIVS
1655 PDTCRYPT_STRICTIVS = True
1656 elif arg in [ "-o", "--out", "--dest", "--sink" ]:
1657 outsspec = checked_arg ()
1658 if PDTCRYPT_VERBOSE is True: noise ("PDT: decrypt to %s" % outsspec)
1659 elif arg in [ "-f", "--force" ]:
1660 global PDTCRYPT_OVERWRITE
1661 PDTCRYPT_OVERWRITE = True
1662 if PDTCRYPT_VERBOSE is True: noise ("PDT: overwrite existing files")
1663 elif arg in [ "-S", "--split" ]:
1664 mode |= PDTCRYPT_SPLIT
1665 if PDTCRYPT_VERBOSE is True: noise ("PDT: split files")
1666 elif arg in [ "-D", "--no-decrypt" ]:
1667 mode &= ~PDTCRYPT_DECRYPT
1668 if PDTCRYPT_VERBOSE is True: noise ("PDT: not decrypting")
1669 elif arg in [ "-k", "--key" ]:
1670 arg = checked_arg ()
1671 checked_secret (PDTCRYPT_SECRET_KEY, arg)
1672 if PDTCRYPT_VERBOSE is True: noise ("PDT: decrypting with key")
1674 bail ("ERROR: unexpected positional argument “%s”" % arg)
1675 elif subcommand == PDTCRYPT_SUB_SCRYPT:
1676 if arg in [ "-n", "--nacl", "--salt" ]:
1677 nacl = checked_arg ()
1678 if PDTCRYPT_VERBOSE is True: noise ("PDT: salt key with %s" % nacl)
1679 elif arg in [ "-f", "--format" ]:
1680 arg = checked_arg ()
1682 scrypt_format = PDTCRYPT_SCRYPT_FORMAT [arg]
1684 bail ("ERROR: invalid scrypt output format %s" % arg)
1685 if PDTCRYPT_VERBOSE is True:
1686 noise ("PDT: scrypt output format “%s”" % scrypt_format)
1688 bail ("ERROR: unexpected positional argument “%s”" % arg)
1691 if PDTCRYPT_VERBOSE is True:
1692 noise ("ERROR: no password or key specified, trying $PDTCRYPT_PASSWORD")
1693 epw = os.getenv ("PDTCRYPT_PASSWORD")
1695 checked_secret (PDTCRYPT_SECRET_PW, epw.strip ())
1698 if PDTCRYPT_VERBOSE is True:
1699 noise ("ERROR: no password or key specified, trying $PDTCRYPT_KEY")
1700 ek = os.getenv ("PDTCRYPT_KEY")
1702 checked_secret (PDTCRYPT_SECRET_KEY, ek.strip ())
1705 if subcommand == PDTCRYPT_SUB_SCRYPT:
1706 bail ("ERROR: scrypt hash mode requested but no password given")
1707 elif mode & PDTCRYPT_DECRYPT:
1708 bail ("ERROR: encryption requested but no password given")
1710 if subcommand == PDTCRYPT_SUB_SCRYPT:
1711 if secret [0] == PDTCRYPT_SECRET_KEY:
1712 bail ("ERROR: scrypt mode requires a password")
1713 if insspec is not None and nacl is not None \
1714 or insspec is None and nacl is None :
1715 bail ("ERROR: please supply either an input file or "
1720 if insspec is not None or subcommand != PDTCRYPT_SUB_SCRYPT:
1721 ins = deptdcrypt_mk_stream (PDTCRYPT_SOURCE, insspec or "-")
1723 if subcommand == PDTCRYPT_SUB_SCRYPT:
1724 return True, partial (mode_scrypt, secret [1].encode (), ins, nacl,
1727 if mode & PDTCRYPT_SPLIT: # destination must be directory
1728 if outsspec is None or outsspec == "-":
1729 bail ("ERROR: split mode is incompatible with stdout sink")
1733 os.makedirs (outsspec, 0o700)
1734 except FileExistsError:
1735 # if it’s a directory with appropriate perms, everything is
1736 # good; otherwise, below invocation of open(2) will fail
1738 outs = os.open (outsspec, os.O_DIRECTORY, 0o600)
1739 except FileNotFoundError as exn:
1740 bail ("ERROR: cannot create target directory “%s”" % outsspec)
1741 except NotADirectoryError as exn:
1742 bail ("ERROR: target path “%s” is not a directory" % outsspec)
1745 outs = deptdcrypt_mk_stream (PDTCRYPT_SINK, outsspec or "-")
1747 return True, partial (mode_depdtcrypt, mode, secret, ins, outs)
1751 ok, runner = parse_argv (argv)
1753 if ok is True: return runner ()
1758 if __name__ == "__main__":
1759 sys.exit (main (sys.argv))