6 ===============================================================================
7 crypto -- Encryption Layer for the Deltatar Backup
8 ===============================================================================
12 - AES-GCM for the symmetric encryption;
17 - NIST Recommendation for Block Cipher Modes of Operation: Galois/Counter
19 http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf
22 https://cryptome.org/2014/01/aes-gcm-v1.pdf
24 - Authentication weaknesses in GCM
25 http://csrc.nist.gov/groups/ST/toolkit/BCM/documents/comments/CWC-GCM/Ferguson2.pdf
27 Trouble with python-cryptography packages: authentication tags can only be
28 passed in advance: https://github.com/pyca/cryptography/pull/3421
31 -------------------------------------------------------------------------------
33 Errors fall into roughly three categories:
35 - Cryptographical errors or invalid data.
37 - ``InvalidGCMTag`` (decryption failed on account of an invalid GCM
39 - ``InvalidIVFixedPart`` (IV fixed part of object not found in list),
40 - ``DuplicateIV`` (the IV of an encrypted object already occurred),
41 - ``DecryptionError`` (used in CLI decryption for presenting error
42 conditions to the user).
44 - Incorrect usage of the library.
46 - ``InvalidParameter`` (non-conforming user supplied parameter),
47 - ``InvalidHeader`` (data passed for reading not parsable into header),
48 - ``FormatError`` (cannot handle header or parameter version),
51 - Bad internal state. If one of these is encountered it means that a state
52 was reached that shouldn’t occur during normal processing.
57 Also, ``EndOfFile`` is used as a sentinel to communicate that a stream supplied
58 for reading is exhausted.
60 Initialization Vectors
61 -------------------------------------------------------------------------------
63 Initialization vectors are checked reuse during the lifetime of a decryptor.
64 The fixed counters for metadata files cannot be reused and attempts to do so
65 will cause a DuplicateIV error. This means the length of objects encrypted with
66 a metadata counter is capped at 63 GB.
68 For ordinary, non-metadata payload, there is an optional mode with strict IV
69 checking that causes a crypto context to fail if an IV encountered or created
70 was already used for decrypting or encrypting, respectively, an earlier object.
71 Note that this mode can trigger false positives when decrypting non-linearly,
72 e. g. when traversing the same object multiple times. Since the crypto context
73 has no notion of a position in a PDT encrypted archive, this condition must be
74 sorted out downstream.
77 -------------------------------------------------------------------------------
79 ``crypto.py`` may be invoked as a script for decrypting, validating, and
80 splitting PDT encrypted files. Consult the usage message for details.
84 Decrypt from stdin using the password ‘foo’: ::
86 $ crypto.py process foo -i - -o - <some-file.tar.gz.pdtcrypt >some-file.tar.gz
88 Output verbose information about the encrypted objects in the archive: ::
90 $ crypto.py process foo -v -i some-file.tar.gz.pdtcrypt -o /dev/null
91 PDT: decrypt from some-file.tar.gz.pdtcrypt
92 PDT: decrypt to /dev/null
93 PDT: source: file some-file.tar.gz.pdtcrypt
94 PDT: sink: file /dev/null
96 PDT: · version = 1 : 0100
97 PDT: · paramversion = 1 : 0100
98 PDT: · nacl : d270 b031 00d1 87e2 c946 610d 7b7f 7e5f
99 PDT: · iv : 02ee 3dd7 a963 1eb1 0100 0000
100 PDT: · ctsize = 591 : 4f02 0000 0000 0000
101 PDT: · tag : 5b2d 6d8b 8f82 4842 12fd 0b10 b6e3 369b
102 PDT: 64 decrypt obj no. 1, 591 B
103 PDT: · [64] 0% done, read block (591 B of 591 B remaining)
104 PDT: · decrypt ciphertext 591 B
105 PDT: · decrypt plaintext 591 B
109 Also, the mode *scrypt* allows deriving encryption keys. To calculate the
110 encryption key from the password ‘foo’ and the salt of the first object in a
111 PDT encrypted file: ::
113 $ crypto.py scrypt foo -i some-file.pdtcrypt
114 {"paramversion": 1, "salt": "Cqzbk48e3peEjzWto8D0yA==", "key": "JH9EkMwaM4x9F5aim5gK/Q=="}
116 The computed 16 byte key is given in hexadecimal notation in the value to
117 ``hash`` and can be fed into Python’s ``binascii.unhexlify()`` to obtain the
118 corresponding binary representation.
120 Note that in Scrypt hashing mode, no data integrity checks are being performed.
121 If the wrong password is given, a wrong key will be derived. Whether the password
122 was indeed correct can only be determined by decrypting. Note that since PDT
123 archives essentially consist of a stream of independent objects, the salt and
124 other parameters may change. Thus a key derived using above method from the
125 first object doesn’t necessarily apply to any of the subsequent objects.
134 from functools import reduce, partial
143 except ImportError as exn:
146 if __name__ == "__main__": ## Work around the import mechanism’s lest Python’s
147 pwd = os.getcwd() ## preference for local imports causes a cyclical
148 ## import (crypto → pylibscrypt → […] → ./tarfile → crypto).
149 sys.path = [ p for p in sys.path if p.find ("deltatar") < 0 ]
152 from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
153 from cryptography.hazmat.backends import default_backend
157 __all__ = [ "hdr_make", "hdr_read", "hdr_fmt", "hdr_fmt_pretty"
159 , "PDTCRYPT_HDR_SIZE", "AES_GCM_IV_CNT_DATA"
160 , "AES_GCM_IV_CNT_INFOFILE", "AES_GCM_IV_CNT_INDEX"
164 ###############################################################################
166 ###############################################################################
168 class EndOfFile (Exception):
172 def __init__ (self, n=None, msg=None):
178 class InvalidParameter (Exception):
179 """Inputs not valid for PDT encryption."""
183 class InvalidHeader (Exception):
184 """Header not valid."""
188 class InvalidGCMTag (Exception):
190 The GCM tag calculated during decryption differs from that in the object
196 class InvalidIVFixedPart (Exception):
198 IV fixed part not in supplied list: either the backup is corrupt or the
199 current object does not belong to it.
204 class IVFixedPartError (Exception):
206 Error creating a unique IV fixed part: repeated calls to system RNG yielded
207 the same sequence of bytes as the last IV used.
212 class InvalidFileCounter (Exception):
214 When encrypting, an attempted reuse of a dedicated counter (info file,
215 index file) was caught.
220 class DuplicateIV (Exception):
222 During encryption, the current IV fixed part is identical to an already
223 existing IV (same prefix and file counter). This indicates tampering or
224 programmer error and cannot be recovered from.
229 class NonConsecutiveIV (Exception):
231 IVs not numbered consecutively. This is a hard error with strict IV
232 checking. Precludes random access to the encrypted objects.
237 class FormatError (Exception):
238 """Unusable parameters in header."""
242 class DecryptionError (Exception):
243 """Error during decryption with ``crypto.py`` on the command line."""
247 class Unreachable (Exception):
249 Makeshift __builtin_unreachable(); always a programmer error if
255 class InternalError (Exception):
256 """Errors not ascribable to bad user inputs or cryptography."""
260 ###############################################################################
261 ## crypto layer version
262 ###############################################################################
264 ENCRYPTION_PARAMETERS = \
266 { "kdf": ("dummy", 16)
267 , "enc": "passthrough" }
275 , "enc": "aes-gcm" } }
277 ###############################################################################
279 ###############################################################################
281 PDTCRYPT_HDR_MAGIC = b"PDTCRYPT"
283 PDTCRYPT_HDR_SIZE_MAGIC = 8 # 8
284 PDTCRYPT_HDR_SIZE_VERSION = 2 # 10
285 PDTCRYPT_HDR_SIZE_PARAMVERSION = 2 # 12
286 PDTCRYPT_HDR_SIZE_NACL = 16 # 28
287 PDTCRYPT_HDR_SIZE_IV = 12 # 40
288 PDTCRYPT_HDR_SIZE_CTSIZE = 8 # 48
289 PDTCRYPT_HDR_SIZE_TAG = 16 # 64 GCM auth tag
291 PDTCRYPT_HDR_SIZE = PDTCRYPT_HDR_SIZE_MAGIC + PDTCRYPT_HDR_SIZE_VERSION \
292 + PDTCRYPT_HDR_SIZE_PARAMVERSION + PDTCRYPT_HDR_SIZE_NACL \
293 + PDTCRYPT_HDR_SIZE_IV + PDTCRYPT_HDR_SIZE_CTSIZE \
294 + PDTCRYPT_HDR_SIZE_TAG # = 64
296 # precalculate offsets since Python can’t do constant folding over names
297 HDR_OFF_VERSION = PDTCRYPT_HDR_SIZE_MAGIC
298 HDR_OFF_PARAMVERSION = HDR_OFF_VERSION + PDTCRYPT_HDR_SIZE_VERSION
299 HDR_OFF_NACL = HDR_OFF_PARAMVERSION + PDTCRYPT_HDR_SIZE_PARAMVERSION
300 HDR_OFF_IV = HDR_OFF_NACL + PDTCRYPT_HDR_SIZE_NACL
301 HDR_OFF_CTSIZE = HDR_OFF_IV + PDTCRYPT_HDR_SIZE_IV
302 HDR_OFF_TAG = HDR_OFF_CTSIZE + PDTCRYPT_HDR_SIZE_CTSIZE
306 FMT_I2N_IV = "<8sL" # 8 random bytes ‖ 32 bit counter
307 FMT_I2N_HDR = ("<" # host byte order
311 "16s" # sodium chloride
317 AES_GCM_MAX_SIZE = (1 << 36) - (1 << 5) # 2^39 - 2^8 b ≅ 64 GB
318 PDTCRYPT_MAX_OBJ_SIZE_DEFAULT = 63 * (1 << 30) # 63 GB
319 PDTCRYPT_MAX_OBJ_SIZE = PDTCRYPT_MAX_OBJ_SIZE_DEFAULT
321 # index and info files are written on-the fly while encrypting so their
322 # counters must be available inadvance
323 AES_GCM_IV_CNT_INFOFILE = 1 # constant
324 AES_GCM_IV_CNT_INDEX = AES_GCM_IV_CNT_INFOFILE + 1
325 AES_GCM_IV_CNT_DATA = AES_GCM_IV_CNT_INDEX + 1 # also for multivolume
326 AES_GCM_IV_CNT_MAX_DEFAULT = 0xffFFffFF
327 AES_GCM_IV_CNT_MAX = AES_GCM_IV_CNT_MAX_DEFAULT
329 # IV structure and generation
330 PDTCRYPT_IV_GEN_MAX_RETRIES = 10 # ×
331 PDTCRYPT_IV_FIXEDPART_SIZE = 8 # B
332 PDTCRYPT_IV_COUNTER_SIZE = 4 # B
334 ###############################################################################
336 ###############################################################################
342 # , paramversion : u16
348 # fn hdr_read (f : handle) -> hdrinfo;
349 # fn hdr_make (f : handle, h : hdrinfo) -> IOResult<usize>;
350 # fn hdr_fmt (h : hdrinfo) -> String;
355 Read bytes as header structure.
357 If the input could not be interpreted as a header, fail with
362 mag, version, paramversion, nacl, iv, ctsize, tag = \
363 struct.unpack (FMT_I2N_HDR, data)
364 except Exception as exn:
365 raise InvalidHeader ("error unpacking header from [%r]: %s"
366 % (binascii.hexlify (data), str (exn)))
368 if mag != PDTCRYPT_HDR_MAGIC:
369 raise InvalidHeader ("bad magic in header: expected [%s], got [%s]"
370 % (PDTCRYPT_HDR_MAGIC, mag))
373 { "version" : version
374 , "paramversion" : paramversion
382 def hdr_read_stream (instr):
384 Read header from stream at the current position.
386 Fail with ``InvalidHeader`` if insufficient bytes were read from the
387 stream, or if the content could not be interpreted as a header.
389 data = instr.read(PDTCRYPT_HDR_SIZE)
393 elif ldata != PDTCRYPT_HDR_SIZE:
394 raise InvalidHeader ("hdr_read_stream: expected %d B, received %d B"
395 % (PDTCRYPT_HDR_SIZE, ldata))
396 return hdr_read (data)
399 def hdr_from_params (version, paramversion, nacl, iv, ctsize, tag):
401 Assemble the necessary values into a PDTCRYPT header.
403 :type version: int to fit uint16_t
404 :type paramversion: int to fit uint16_t
405 :type nacl: bytes to fit uint8_t[16]
406 :type iv: bytes to fit uint8_t[12]
407 :type size: int to fit uint64_t
408 :type tag: bytes to fit uint8_t[16]
410 buf = bytearray (PDTCRYPT_HDR_SIZE)
411 bufv = memoryview (buf)
414 struct.pack_into (FMT_I2N_HDR, bufv, 0,
416 version, paramversion, nacl, iv, ctsize, tag)
417 except Exception as exn:
418 return False, "error assembling header: %s" % str (exn)
420 return True, bytes (buf)
423 def hdr_make_dummy (s):
425 Create a header sized block of bytes initialized to a value derived from a
426 string. Used to verify we’ve jumped back correctly to the actual position
427 of the object header.
429 c = reduce (lambda a, c: a + ord(c), s, 0) % 0xFF
430 return bytes (bytearray (struct.pack ("B", c)) * PDTCRYPT_HDR_SIZE)
435 Assemble a header from the given header structure.
437 return hdr_from_params (version=hdr.get("version"),
438 paramversion=hdr.get("paramversion"),
439 nacl=hdr.get("nacl"), iv=hdr.get("iv"),
440 ctsize=hdr.get("ctsize"), tag=hdr.get("tag"))
443 HDR_FMT = "I2n_header { version: %d, paramversion: %d, nacl: %s[%d]," \
444 " iv: %s[%d], ctsize: %d, tag: %s[%d] }"
447 """Format a header structure into readable output."""
448 return HDR_FMT % (h["version"], h["paramversion"],
449 binascii.hexlify (h["nacl"]), len(h["nacl"]),
450 binascii.hexlify (h["iv"]), len(h["iv"]),
452 binascii.hexlify (h["tag"]), len(h["tag"]))
455 def hex_spaced_of_bytes (b):
456 """Format bytes object, hexdump style."""
457 return " ".join ([ "%.2x%.2x" % (c1, c2)
458 for c1, c2 in zip (b[0::2], b[1::2]) ]) \
459 + (len (b) | 1 == len (b) and " %.2x" % b[-1] or "") # odd lengths
462 def hdr_iv_counter (h):
463 """Extract the variable part of the IV of the given header."""
464 _fixed, cnt = struct.unpack (FMT_I2N_IV, h ["iv"])
468 def hdr_iv_fixed (h):
469 """Extract the fixed part of the IV of the given header."""
470 fixed, _cnt = struct.unpack (FMT_I2N_IV, h ["iv"])
474 hdr_dump = hex_spaced_of_bytes
478 """version = %-4d : %s
479 paramversion = %-4d : %s
486 def hdr_fmt_pretty (h):
488 Format header structure into multi-line representation of its contents and
489 their raw representation. (Omit the implicit “PDTCRYPT” magic bytes that
490 precede every header.)
492 return HDR_FMT_PRETTY \
494 hex_spaced_of_bytes (struct.pack (FMT_UINT16_LE, h["version"])),
496 hex_spaced_of_bytes (struct.pack (FMT_UINT16_LE, h["paramversion"])),
497 hex_spaced_of_bytes (h["nacl"]),
498 hex_spaced_of_bytes (h["iv"]),
500 hex_spaced_of_bytes (struct.pack (FMT_UINT64_LE, h["ctsize"])),
501 hex_spaced_of_bytes (h["tag"]))
503 IV_FMT = "((f %s) (c %d))"
506 """Format the two components of an IV in a readable fashion."""
507 fixed, cnt = struct.unpack (FMT_I2N_IV, iv)
508 return IV_FMT % (binascii.hexlify (fixed), cnt)
511 ###############################################################################
513 ###############################################################################
515 class Location (object):
519 def restore_loc_fmt (loc):
521 % (loc.n, loc.offset)
523 def locate_hdr_candidates (fd):
525 Walk over instances of the magic string in the payload, collecting their
526 positions. If the offset of the first found instance is not zero, the file
527 begins with leading garbage.
529 :return: The list of offsets in the file.
533 mm = mmap.mmap(fd, 0, mmap.MAP_SHARED, mmap.PROT_READ)
536 pos = mm.find (PDTCRYPT_HDR_MAGIC, pos)
545 ###############################################################################
546 ## passthrough / null encryption
547 ###############################################################################
549 class PassthroughCipher (object):
551 tag = struct.pack ("<QQ", 0, 0)
553 def __init__ (self) : pass
555 def update (self, b) : return b
557 def finalize (self) : return b""
559 def finalize_with_tag (self, _) : return b""
561 ###############################################################################
562 ## convenience wrapper
563 ###############################################################################
566 def kdf_dummy (klen, password, _nacl):
568 Fake KDF for testing purposes that is called when parameter version zero is
571 q, r = divmod (klen, len (password))
572 if isinstance (password, bytes) is False:
573 password = password.encode ()
574 return password * q + password [:r], b""
577 SCRYPT_KEY_MEMO = { } # static because needed for both the info file and the archive
580 def kdf_scrypt (params, password, nacl):
582 Wrapper for the Scrypt KDF, corresponds to parameter version one. The
583 computation result is memoized based on the inputs to facilitate spawning
584 multiple encryption contexts.
589 dkLen = params["dkLen"]
592 nacl = os.urandom (params["NaCl_LEN"])
594 key_parms = (password, nacl, N, r, p, dkLen)
595 global SCRYPT_KEY_MEMO
596 if key_parms not in SCRYPT_KEY_MEMO:
597 SCRYPT_KEY_MEMO [key_parms] = \
598 pylibscrypt.scrypt (password, nacl, N, r, p, dkLen)
599 return SCRYPT_KEY_MEMO [key_parms], nacl
602 def kdf_by_version (paramversion=None, defs=None):
604 Pick the KDF handler corresponding to the parameter version or the
607 :rtype: function (password : str, nacl : str) -> str
609 if paramversion is not None:
610 defs = ENCRYPTION_PARAMETERS.get(paramversion, None)
612 raise InvalidParameter ("no encryption parameters for version %r"
614 (kdf, params) = defs["kdf"]
616 if kdf == "scrypt" : fn = kdf_scrypt
617 if kdf == "dummy" : fn = kdf_dummy
619 raise ValueError ("key derivation method %r unknown" % kdf)
620 return partial (fn, params)
623 ###############################################################################
625 ###############################################################################
627 def scrypt_hashsource (pw, ins):
629 Calculate the SCRYPT hash from the password and the information contained
630 in the first header found in ``ins``.
632 This does not validate whether the first object is encrypted correctly.
634 if isinstance (pw, str) is True:
636 elif isinstance (pw, bytes) is False:
637 raise InvalidParameter ("password must be a string, not %s"
639 if isinstance (ins, io.BufferedReader) is False and \
640 isinstance (ins, io.FileIO) is False:
641 raise InvalidParameter ("file to hash must be opened in “binary” mode")
644 hdr = hdr_read_stream (ins)
645 except EndOfFile as exn:
646 noise ("PDT: malformed input: end of file reading first object header")
651 pver = hdr ["paramversion"]
652 if PDTCRYPT_VERBOSE is True:
653 noise ("PDT: salt of first object : %s" % binascii.hexlify (nacl))
654 noise ("PDT: parameter version of archive : %d" % pver)
657 defs = ENCRYPTION_PARAMETERS.get(pver, None)
658 kdfname, params = defs ["kdf"]
659 if kdfname != "scrypt":
660 noise ("PDT: input is not an SCRYPT archive")
663 kdf = kdf_by_version (None, defs)
664 except ValueError as exn:
665 noise ("PDT: object has unknown parameter version %d" % pver)
667 hsh, _void = kdf (pw, nacl)
669 return hsh, nacl, hdr ["version"], pver
672 def scrypt_hashfile (pw, fname):
674 Calculate the SCRYPT hash from the password and the information contained
675 in the first header found in the given file. The header is read only at
678 with deptdcrypt_mk_stream (PDTCRYPT_SOURCE, fname or "-") as ins:
679 hsh, _void, _void, _void = scrypt_hashsource (pw, ins)
683 ###############################################################################
685 ###############################################################################
687 class Crypto (object):
689 Encryption context to remain alive throughout an entire tarfile pass.
694 cnt = None # file counter (uint32_t != 0)
695 iv = None # current IV
696 fixed = None # accu for 64 bit fixed parts of IV
697 used_ivs = None # tracks IVs
698 strict_ivs = False # if True, panic on duplicate object IV
707 info_counter_used = False
708 index_counter_used = False
710 def __init__ (self, *al, **akv):
711 self.used_ivs = set ()
712 self.set_parameters (*al, **akv)
715 def next_fixed (self):
720 def set_object_counter (self, cnt=None):
722 Safely set the internal counter of encrypted objects. Numerous
725 The same counter may not be reused in combination with one IV fixed
726 part. This is validated elsewhere in the IV handling.
728 Counter zero is invalid. The first two counters are reserved for
729 metadata. The implementation does not allow for splitting metadata
730 files over multiple encrypted objects. (This would be possible by
731 assigning new fixed parts.) Thus in a Deltatar backup there is at most
732 one object with a counter value of one and two. On creation of a
733 context, the initial counter may be chosen. The globals
734 ``AES_GCM_IV_CNT_INFOFILE`` and ``AES_GCM_IV_CNT_INDEX`` can be used to
735 request one of the reserved values. If one of these values has been
736 used, any further attempt of setting the counter to that value will
737 be rejected with an ``InvalidFileCounter`` exception.
739 Out of bounds values (i. e. below one and more than the maximum of 2³²)
740 cause an ``InvalidParameter`` exception to be thrown.
743 self.cnt = AES_GCM_IV_CNT_DATA
745 if cnt == 0 or cnt > AES_GCM_IV_CNT_MAX + 1:
746 raise InvalidParameter ("invalid counter value %d requested: "
747 "acceptable values are from 1 to %d"
748 % (cnt, AES_GCM_IV_CNT_MAX))
749 if cnt == AES_GCM_IV_CNT_INFOFILE:
750 if self.info_counter_used is True:
751 raise InvalidFileCounter ("attempted to reuse info file "
752 "counter %d: must be unique" % cnt)
753 self.info_counter_used = True
754 elif cnt == AES_GCM_IV_CNT_INDEX:
755 if self.index_counter_used is True:
756 raise InvalidFileCounter ("attempted to reuse index file "
757 " counter %d: must be unique" % cnt)
758 self.index_counter_used = True
759 if cnt <= AES_GCM_IV_CNT_MAX:
762 # cnt == AES_GCM_IV_CNT_MAX + 1 → wrap
763 self.cnt = AES_GCM_IV_CNT_DATA
767 def set_parameters (self, password=None, key=None, paramversion=None,
768 nacl=None, counter=None, strict_ivs=False):
770 Configure the internal state of a crypto context. Not intended for
774 self.set_object_counter (counter)
775 self.strict_ivs = strict_ivs
777 if paramversion is not None:
778 self.paramversion = paramversion
781 self.key, self.nacl = key, nacl
784 if password is not None:
785 if isinstance (password, bytes) is False:
786 password = str.encode (password)
787 self.password = password
788 if paramversion is None and nacl is None:
789 # postpone key setup until first header is available
791 kdf = kdf_by_version (paramversion)
793 self.key, self.nacl = kdf (password, nacl)
796 def process (self, buf):
798 Encrypt / decrypt a buffer. Invokes the ``.update()`` method on the
799 wrapped encryptor or decryptor, respectively.
801 The Cryptography exception ``AlreadyFinalized`` is translated to an
802 ``InternalError`` at this point. It may occur in sound code when the GC
803 closes an encrypting stream after an error. Everywhere else it must be
807 raise RuntimeError ("process: context not initialized")
808 self.stats ["in"] += len (buf)
810 out = self.enc.update (buf)
811 except cryptography.exceptions.AlreadyFinalized as exn:
812 raise InternalError (exn)
813 self.stats ["out"] += len (out)
817 def next (self, password, paramversion, nacl, iv):
819 Prepare for encrypting another object: Reset the data counters and
820 change the configuration in case one of the variable parameters differs
821 from the last object. Also check the IV for duplicates and error out
822 if strict checking was requested.
826 self.stats ["obj"] += 1
828 self.check_duplicate_iv (iv)
830 if ( self.paramversion != paramversion
831 or self.password != password
832 or self.nacl != nacl):
833 self.set_parameters (password=password, paramversion=paramversion,
834 nacl=nacl, strict_ivs=self.strict_ivs)
837 def check_duplicate_iv (self, iv):
839 Add an IV (the 12 byte representation as in the header) to the list. With
840 strict checking enabled, this will throw a ``DuplicateIV``. Depending on
841 the context, this may indicate a serious error (IV reuse).
843 if self.strict_ivs is True and iv in self.used_ivs:
844 raise DuplicateIV ("iv %s was reused" % iv_fmt (iv))
845 # vi has not been used before; add to collection
846 self.used_ivs.add (iv)
851 Access the data counters.
853 return self.stats ["obj"], self.stats ["in"], self.stats ["out"]
858 Clear the current context regardless of its finalization state. The
859 next operation must be ``.next()``.
864 class Encrypt (Crypto):
870 def __init__ (self, version, paramversion, password=None, key=None, nacl=None,
871 counter=AES_GCM_IV_CNT_DATA, strict_ivs=True):
873 The ctor will throw immediately if one of the parameters does not conform
876 counter=AES_GCM_IV_CNT_DATA, strict_ivs=True):
877 :type version: int to fit uint16_t
878 :type paramversion: int to fit uint16_t
879 :param password: mutually exclusive with ``key``
880 :type password: bytes
881 :param key: mutually exclusive with ``password``
884 :type counter: initial object counter the values
885 ``AES_GCM_IV_CNT_INFOFILE`` and
886 ``AES_GCM_IV_CNT_INDEX`` are unique in each backup set
887 and cannot be reused even with different fixed parts.
888 :type strict_ivs: bool
890 if password is None and key is None \
891 or password is not None and key is not None :
892 raise InvalidParameter ("__init__: need either key or password")
895 if isinstance (key, bytes) is False:
896 raise InvalidParameter ("__init__: key must be provided as "
897 "bytes, not %s" % type (key))
899 raise InvalidParameter ("__init__: salt must be provided along "
900 "with encryption key")
901 else: # password, no key
902 if isinstance (password, str) is False:
903 raise InvalidParameter ("__init__: password must be a string, not %s"
905 if len (password) == 0:
906 raise InvalidParameter ("__init__: supplied empty password but not "
907 "permitted for PDT encrypted files")
909 if isinstance (version, int) is False:
910 raise InvalidParameter ("__init__: version number must be an "
911 "integer, not %s" % type (version))
913 raise InvalidParameter ("__init__: version number must be a "
914 "nonnegative integer, not %d" % version)
916 if isinstance (paramversion, int) is False:
917 raise InvalidParameter ("__init__: crypto parameter version number "
918 "must be an integer, not %s"
919 % type (paramversion))
921 raise InvalidParameter ("__init__: crypto parameter version number "
922 "must be a nonnegative integer, not %d"
926 if isinstance (nacl, bytes) is False:
927 raise InvalidParameter ("__init__: salt given, but of type %s "
928 "instead of bytes" % type (nacl))
929 # salt length would depend on the actual encryption so it can’t be
930 # validated at this point
932 self.version = version
933 self.paramenc = ENCRYPTION_PARAMETERS.get (paramversion) ["enc"]
935 super().__init__ (password, key, paramversion, nacl, counter=counter,
936 strict_ivs=strict_ivs)
939 def next_fixed (self, retries=PDTCRYPT_IV_GEN_MAX_RETRIES):
941 Generate the next IV fixed part by reading eight bytes from
942 ``/dev/urandom``. The buffer so obtained is tested against the fixed
943 parts used so far to prevent accidental reuse of IVs. After a
944 configurable number of attempts to create a unique fixed part, it will
945 refuse to continue with an ``IVFixedPartError``. This is unlikely to
946 ever happen on a normal system but may detect an issue with the random
949 The list of fixed parts that were used by the context at hand can be
950 accessed through the ``.fixed`` list. Its last element is the fixed
951 part currently in use.
955 fp = os.urandom (PDTCRYPT_IV_FIXEDPART_SIZE)
956 if fp not in self.fixed:
957 self.fixed.append (fp)
960 raise IVFixedPartError ("error obtaining a unique IV fixed part from "
961 "/dev/urandom; giving up after %d tries" % i)
966 Construct a 12-bytes IV from the current fixed part and the object
969 return struct.pack(FMT_I2N_IV, self.fixed [-1], self.cnt)
972 def next (self, filename=None, counter=None):
974 Prepare for encrypting the next incoming object. Update the counter
975 and put together the IV, possibly changing prefixes. Then create the
978 The argument ``counter`` can be used to specify a file counter for this
979 object. Unless it is one of the reserved values, the counter of
980 subsequent objects will be computed from this one.
982 If this is the first object in a series, ``filename`` is required,
983 otherwise it is reused if not present. The value is used to derive a
984 header sized placeholder to use until after encryption when all the
985 inputs to construct the final header are available. This is then
986 matched in ``.done()`` against the value found at the position of the
987 header. The motivation for this extra check is primarily to assist
988 format debugging: It makes stray headers easy to spot in malformed
992 if self.lastinfo is None:
993 raise InvalidParameter ("next: filename is mandatory for "
995 filename, _dummy = self.lastinfo
997 if isinstance (filename, str) is False:
998 raise InvalidParameter ("next: filename must be a string, no %s"
1000 if counter is not None:
1001 if isinstance (counter, int) is False:
1002 raise InvalidParameter ("next: the supplied counter is of "
1003 "invalid type %s; please pass an "
1004 "integer instead" % type (counter))
1005 self.set_object_counter (counter)
1007 self.iv = self.iv_make ()
1008 if self.paramenc == "aes-gcm":
1010 ( algorithms.AES (self.key)
1011 , modes.GCM (self.iv)
1012 , backend = default_backend ()) \
1014 elif self.paramenc == "passthrough":
1015 self.enc = PassthroughCipher ()
1017 raise InvalidParameter ("next: parameter version %d not known"
1018 % self.paramversion)
1019 hdrdum = hdr_make_dummy (filename)
1020 self.lastinfo = (filename, hdrdum)
1021 super().next (self.password, self.paramversion, self.nacl, self.iv)
1023 self.set_object_counter (self.cnt + 1)
1027 def done (self, cmpdata):
1029 Complete encryption of an object. After this has been called, attempts
1030 of encrypting further data will cause an error until ``.next()`` is
1033 Returns a 64 bytes buffer containing the object header including all
1034 values including the “late” ones e. g. the ciphertext size and the
1037 if isinstance (cmpdata, bytes) is False:
1038 raise InvalidParameter ("done: comparison input expected as bytes, "
1039 "not %s" % type (cmpdata))
1040 if self.lastinfo is None:
1041 raise RuntimeError ("done: encryption context not initialized")
1042 filename, hdrdum = self.lastinfo
1043 if cmpdata != hdrdum:
1044 raise RuntimeError ("done: bad sync of header for object %d: "
1045 "preliminary data does not match; this likely "
1046 "indicates a wrongly repositioned stream"
1048 data = self.enc.finalize ()
1049 self.stats ["out"] += len (data)
1050 self.ctsize += len (data)
1051 ok, hdr = hdr_from_params (self.version, self.paramversion, self.nacl,
1052 self.iv, self.ctsize, self.enc.tag)
1054 raise InternalError ("error constructing header: %r" % hdr)
1055 return data, hdr, self.fixed
1058 def process (self, buf):
1060 Encrypt a chunk of plaintext with the active encryptor. Returns the
1061 size of the input consumed. This **must** be checked downstream. If the
1062 maximum possible object size has been reached, the current context must
1063 be finalized and a new one established before any further data can be
1064 encrypted. The second argument is the remainder of the plaintext that
1065 was not encrypted for the caller to use immediately after the new
1068 if isinstance (buf, bytes) is False:
1069 raise InvalidParameter ("process: expected byte buffer, not %s"
1072 newptsize = self.ptsize + bsize
1073 diff = newptsize - PDTCRYPT_MAX_OBJ_SIZE
1076 newptsize = PDTCRYPT_MAX_OBJ_SIZE
1077 self.ptsize = newptsize
1078 data = super().process (buf [:bsize])
1079 self.ctsize += len (data)
1083 class Decrypt (Crypto):
1085 tag = None # GCM tag, part of header
1086 last_iv = None # check consecutive ivs in strict mode
1088 def __init__ (self, password=None, key=None, counter=None, fixedparts=None,
1091 Sanitizing ctor for the decryption context. ``fixedparts`` specifies a
1092 list of IV fixed parts accepted during decryption. If a fixed part is
1093 encountered that is not in the list, decryption will fail.
1095 :param password: mutually exclusive with ``key``
1096 :type password: bytes
1097 :param key: mutually exclusive with ``password``
1099 :type counter: initial object counter the values
1100 ``AES_GCM_IV_CNT_INFOFILE`` and
1101 ``AES_GCM_IV_CNT_INDEX`` are unique in each backup set
1102 and cannot be reused even with different fixed parts.
1103 :type fixedparts: bytes list
1105 if password is None and key is None \
1106 or password is not None and key is not None :
1107 raise InvalidParameter ("__init__: need either key or password")
1110 if isinstance (key, bytes) is False:
1111 raise InvalidParameter ("__init__: key must be provided as "
1112 "bytes, not %s" % type (key))
1113 else: # password, no key
1114 if isinstance (password, str) is False:
1115 raise InvalidParameter ("__init__: password must be a string, not %s"
1117 if len (password) == 0:
1118 raise InvalidParameter ("__init__: supplied empty password but not "
1119 "permitted for PDT encrypted files")
1121 if fixedparts is not None:
1122 if isinstance (fixedparts, list) is False:
1123 raise InvalidParameter ("__init__: IV fixed parts must be "
1124 "supplied as list, not %s"
1125 % type (fixedparts))
1126 self.fixed = fixedparts
1129 super().__init__ (password=password, key=key, counter=counter,
1130 strict_ivs=strict_ivs)
1133 def valid_fixed_part (self, iv):
1135 Check if a fixed part was already seen.
1137 # check if fixed part is known
1138 fixed, _cnt = struct.unpack (FMT_I2N_IV, iv)
1139 i = bisect.bisect_left (self.fixed, fixed)
1140 return i != len (self.fixed) and self.fixed [i] == fixed
1143 def check_consecutive_iv (self, iv):
1145 Check whether the counter part of the given IV is indeed the successor
1146 of the currently present counter. This should always be the case for
1147 the objects in a well formed PDT archive but should not be enforced
1148 when decrypting out-of-order.
1150 fixed, cnt = struct.unpack (FMT_I2N_IV, iv)
1151 if self.strict_ivs is True \
1152 and self.last_iv is not None \
1153 and self.last_iv [0] == fixed \
1154 and self.last_iv [1] != cnt - 1:
1155 raise NonConsecutiveIV ("iv %s counter not successor of "
1156 "last object (expected %d, found %d)"
1157 % (iv_fmt (self.last_iv [1]), cnt))
1158 self.last_iv = (iv, cnt)
1161 def next (self, hdr):
1163 Start decrypting the next object. The PDTCRYPT header for the object
1164 can be given either as already parsed object or as bytes.
1166 if isinstance (hdr, bytes) is True:
1167 hdr = hdr_read (hdr)
1168 elif isinstance (hdr, dict) is False:
1169 # this won’t catch malformed specs though
1170 raise InvalidParameter ("next: wrong type of parameter hdr: "
1171 "expected bytes or spec, got %s"
1174 paramversion = hdr ["paramversion"]
1179 raise InvalidHeader ("next: not a header %r" % hdr)
1181 super().next (self.password, paramversion, nacl, iv)
1182 if self.fixed is not None and self.valid_fixed_part (iv) is False:
1183 raise InvalidIVFixedPart ("iv %s has invalid fixed part"
1185 self.check_consecutive_iv (iv)
1188 defs = ENCRYPTION_PARAMETERS.get (paramversion, None)
1190 raise FormatError ("header contains unknown parameter version %d; "
1191 "maybe the file was created by a more recent "
1192 "version of Deltatar" % paramversion)
1194 if enc == "aes-gcm":
1196 ( algorithms.AES (self.key)
1197 , modes.GCM (iv, tag=self.tag)
1198 , backend = default_backend ()) \
1200 elif enc == "passthrough":
1201 self.enc = PassthroughCipher ()
1203 raise InternalError ("encryption parameter set %d refers to unknown "
1204 "mode %r" % (paramversion, enc))
1205 self.set_object_counter (self.cnt + 1)
1208 def done (self, tag=None):
1210 Stop decryption of the current object and finalize it with the active
1211 context. This will throw an *InvalidGCMTag* exception to indicate that
1212 the authentication tag does not match the data. If the tag is correct,
1213 the rest of the plaintext is returned.
1218 data = self.enc.finalize ()
1220 if isinstance (tag, bytes) is False:
1221 raise InvalidParameter ("done: wrong type of parameter "
1222 "tag: expected bytes, got %s"
1224 data = self.enc.finalize_with_tag (self.tag)
1225 except cryptography.exceptions.InvalidTag:
1226 raise InvalidGCMTag ("done: tag mismatch of object %d: %s "
1227 "rejected by finalize ()"
1228 % (self.cnt, binascii.hexlify (self.tag)))
1229 self.ctsize += len (data)
1230 self.stats ["out"] += len (data)
1234 def process (self, buf):
1236 Decrypt the bytes object *buf* with the active decryptor.
1238 if isinstance (buf, bytes) is False:
1239 raise InvalidParameter ("process: expected byte buffer, not %s"
1241 self.ctsize += len (buf)
1242 data = super().process (buf)
1243 self.ptsize += len (data)
1247 ###############################################################################
1249 ###############################################################################
1251 def _patch_global (glob, vow, n=None):
1253 Adapt upper file counter bound for testing IV logic. Completely unsafe.
1255 assert vow == "I am fully aware that this will void my warranty."
1256 r = globals () [glob]
1258 n = globals () [glob + "_DEFAULT"]
1259 globals () [glob] = n
1262 _testing_set_AES_GCM_IV_CNT_MAX = \
1263 partial (_patch_global, "AES_GCM_IV_CNT_MAX")
1265 _testing_set_PDTCRYPT_MAX_OBJ_SIZE = \
1266 partial (_patch_global, "PDTCRYPT_MAX_OBJ_SIZE")
1268 ###############################################################################
1269 ## freestanding invocation
1270 ###############################################################################
1272 PDTCRYPT_SUB_PROCESS = 0
1273 PDTCRYPT_SUB_SCRYPT = 1
1274 PDTCRYPT_SUB_SCAN = 2
1277 { "process" : PDTCRYPT_SUB_PROCESS
1278 , "scrypt" : PDTCRYPT_SUB_SCRYPT
1279 , "scan" : PDTCRYPT_SUB_SCAN }
1281 PDTCRYPT_SECRET_PW = 0
1282 PDTCRYPT_SECRET_KEY = 1
1284 PDTCRYPT_DECRYPT = 1 << 0 # decrypt archive with password
1285 PDTCRYPT_SPLIT = 1 << 1 # split archive into individual objects
1286 PDTCRYPT_HASH = 1 << 2 # output scrypt hash for file and given password
1288 PDTCRYPT_SPLITNAME = "pdtcrypt-object-%d.bin"
1290 PDTCRYPT_VERBOSE = False
1291 PDTCRYPT_STRICTIVS = False
1292 PDTCRYPT_OVERWRITE = False
1293 PDTCRYPT_BLOCKSIZE = 1 << 12
1298 PDTCRYPT_DEFAULT_VER = 1
1299 PDTCRYPT_DEFAULT_PVER = 1
1301 # scrypt hashing output control
1302 PDTCRYPT_SCRYPT_INTRANATOR = 0
1303 PDTCRYPT_SCRYPT_PARAMETERS = 1
1304 PDTCRYPT_SCRYPT_DEFAULT = PDTCRYPT_SCRYPT_INTRANATOR
1306 PDTCRYPT_SCRYPT_FORMAT = \
1307 { "i2n" : PDTCRYPT_SCRYPT_INTRANATOR
1308 , "params" : PDTCRYPT_SCRYPT_PARAMETERS }
1311 class PDTDecryptionError (Exception):
1312 """Decryption failed."""
1314 class PDTSplitError (Exception):
1315 """Decryption failed."""
1318 def noise (*a, **b):
1319 print (file=sys.stderr, *a, **b)
1322 class PassthroughDecryptor (object):
1324 curhdr = None # write current header on first data write
1326 def __init__ (self):
1327 if PDTCRYPT_VERBOSE is True:
1328 noise ("PDT: no encryption; data passthrough")
1330 def next (self, hdr):
1331 ok, curhdr = hdr_make (hdr)
1333 raise PDTDecryptionError ("bad header %r" % hdr)
1334 self.curhdr = curhdr
1337 if self.curhdr is not None:
1341 def process (self, d):
1342 if self.curhdr is not None:
1348 def depdtcrypt (mode, secret, ins, outs):
1350 Remove PDTCRYPT layer from all objects encrypted with the secret. Used on a
1351 Deltatar backup this will yield a (possibly Gzip compressed) tarball.
1353 ctleft = -1 # length of ciphertext to consume
1354 ctcurrent = 0 # total ciphertext of current object
1355 total_obj = 0 # total number of objects read
1356 total_pt = 0 # total plaintext bytes
1357 total_ct = 0 # total ciphertext bytes
1358 total_read = 0 # total bytes read
1359 outfile = None # Python file object for output
1361 if mode & PDTCRYPT_DECRYPT: # decryptor
1363 if ks == PDTCRYPT_SECRET_PW:
1364 decr = Decrypt (password=secret [1], strict_ivs=PDTCRYPT_STRICTIVS)
1365 elif ks == PDTCRYPT_SECRET_KEY:
1366 key = binascii.unhexlify (secret [1])
1367 decr = Decrypt (key=key, strict_ivs=PDTCRYPT_STRICTIVS)
1369 raise InternalError ("‘%d’ does not specify a valid kind of secret"
1372 decr = PassthroughDecryptor ()
1375 """Dummy for non-split mode: output file does not vary."""
1378 if mode & PDTCRYPT_SPLIT:
1379 def nextout (outfile):
1381 We were passed an fd as outs for accessing the destination
1382 directory where extracted archive components are supposed
1387 if PDTCRYPT_VERBOSE is True:
1388 noise ("PDT: no output file to close at this point")
1390 if PDTCRYPT_VERBOSE is True:
1391 noise ("PDT: release output file %r" % outfile)
1392 # cleanup happens automatically by the GC; the next
1393 # line will error out on account of an invalid fd
1396 assert total_obj > 0
1397 fname = PDTCRYPT_SPLITNAME % total_obj
1399 oflags = os.O_CREAT | os.O_WRONLY
1400 if PDTCRYPT_OVERWRITE is True:
1401 oflags |= os.O_TRUNC
1404 outfd = os.open (fname, oflags, 0o600, dir_fd=outs)
1405 if PDTCRYPT_VERBOSE is True:
1406 noise ("PDT: new output file %s → %d" % (fname, outfd))
1407 except FileExistsError as exn:
1408 noise ("PDT: refusing to overwrite existing file %s" % fname)
1410 raise PDTSplitError ("destination file %s already exists"
1413 return os.fdopen (outfd, "wb", closefd=True)
1417 """ESPIPE is normal on non-seekable stdio stream."""
1420 except OSError as exn:
1421 if exn.errno == os.errno.ESPIPE:
1424 def out (pt, outfile):
1428 if PDTCRYPT_VERBOSE is True:
1429 noise ("PDT:\t· decrypt plaintext %d B" % (npt))
1431 nn = outfile.write (pt)
1432 except OSError as exn: # probably ENOSPC
1433 raise DecryptionError ("error (%s)" % exn)
1435 raise DecryptionError ("write aborted after %d of %d B" % (nn, npt))
1439 # current object completed; in a valid archive this marks either
1440 # the start of a new header or the end of the input
1441 if ctleft == 0: # current object requires finalization
1442 if PDTCRYPT_VERBOSE is True:
1443 noise ("PDT: %d finalize" % tell (ins))
1446 except InvalidGCMTag as exn:
1447 raise DecryptionError ("error finalizing object %d (%d B): "
1448 "%r" % (total_obj, len (pt), exn)) \
1451 if PDTCRYPT_VERBOSE is True:
1452 noise ("PDT:\t· object validated")
1454 if PDTCRYPT_VERBOSE is True:
1455 noise ("PDT: %d hdr" % tell (ins))
1457 hdr = hdr_read_stream (ins)
1458 total_read += PDTCRYPT_HDR_SIZE
1459 except EndOfFile as exn:
1460 total_read += exn.remainder
1461 if total_ct + total_obj * PDTCRYPT_HDR_SIZE != total_read:
1462 raise PDTDecryptionError ("ciphertext processed (%d B) plus "
1463 "overhead (%d × %d B) does not match "
1464 "the number of bytes read (%d )"
1465 % (total_ct, total_obj, PDTCRYPT_HDR_SIZE,
1467 # the single good exit
1468 return total_read, total_obj, total_ct, total_pt
1469 except InvalidHeader as exn:
1470 raise PDTDecryptionError ("invalid header at position %d in %r "
1471 "(%s)" % (tell (ins), exn, ins))
1472 if PDTCRYPT_VERBOSE is True:
1473 pretty = hdr_fmt_pretty (hdr)
1474 noise (reduce (lambda a, e: (a + "\n" if a else "") + "PDT:\t· " + e,
1475 pretty.splitlines (), ""))
1476 ctcurrent = ctleft = hdr ["ctsize"]
1480 total_obj += 1 # used in file counter with split mode
1482 # finalization complete or skipped in case of first object in
1483 # stream; create a new output file if necessary
1484 outfile = nextout (outfile)
1486 if PDTCRYPT_VERBOSE is True:
1487 noise ("PDT: %d decrypt obj no. %d, %d B"
1488 % (tell (ins), total_obj, ctleft))
1490 # always allocate a new buffer since python-cryptography doesn’t allow
1491 # passing a bytearray :/
1492 nexpect = min (ctleft, PDTCRYPT_BLOCKSIZE)
1493 if PDTCRYPT_VERBOSE is True:
1494 noise ("PDT:\t· [%d] %d%% done, read block (%d B of %d B remaining)"
1496 100 - ctleft * 100 / (ctcurrent > 0 and ctcurrent or 1),
1498 ct = ins.read (nexpect)
1502 raise EndOfFile (nct,
1503 "hit EOF after %d of %d B in block [%d:%d); "
1504 "%d B ciphertext remaining for object no %d"
1505 % (nct, nexpect, off, off + nexpect, ctleft,
1511 if PDTCRYPT_VERBOSE is True:
1512 noise ("PDT:\t· decrypt ciphertext %d B" % (nct))
1513 pt = decr.process (ct)
1517 def deptdcrypt_mk_stream (kind, path):
1518 """Create stream from file or stdio descriptor."""
1519 if kind == PDTCRYPT_SINK:
1521 if PDTCRYPT_VERBOSE is True: noise ("PDT: sink: stdout")
1522 return sys.stdout.buffer
1524 if PDTCRYPT_VERBOSE is True: noise ("PDT: sink: file %s" % path)
1525 return io.FileIO (path, "w")
1526 if kind == PDTCRYPT_SOURCE:
1528 if PDTCRYPT_VERBOSE is True: noise ("PDT: source: stdin")
1529 return sys.stdin.buffer
1531 if PDTCRYPT_VERBOSE is True: noise ("PDT: source: file %s" % path)
1532 return io.FileIO (path, "r")
1534 raise ValueError ("bogus stream “%s” / %s" % (kind, path))
1537 def mode_depdtcrypt (mode, secret, ins, outs):
1539 total_read, total_obj, total_ct, total_pt = \
1540 depdtcrypt (mode, secret, ins, outs)
1541 except DecryptionError as exn:
1542 noise ("PDT: Decryption failed:")
1544 noise ("PDT: “%s”" % exn)
1546 noise ("PDT: Did you specify the correct key / password?")
1549 except PDTSplitError as exn:
1550 noise ("PDT: Split operation failed:")
1552 noise ("PDT: “%s”" % exn)
1554 noise ("PDT: Hint: target directory should be empty.")
1558 if PDTCRYPT_VERBOSE is True:
1559 noise ("PDT: decryption successful" )
1560 noise ("PDT: %.10d bytes read" % total_read)
1561 noise ("PDT: %.10d objects decrypted" % total_obj )
1562 noise ("PDT: %.10d bytes ciphertext" % total_ct )
1563 noise ("PDT: %.10d bytes plaintext" % total_pt )
1569 def mode_scrypt (pw, ins=None, nacl=None, fmt=PDTCRYPT_SCRYPT_INTRANATOR):
1571 paramversion = PDTCRYPT_DEFAULT_PVER
1573 hsh, nacl, version, paramversion = scrypt_hashsource (pw, ins)
1574 defs = ENCRYPTION_PARAMETERS.get(paramversion, None)
1576 nacl = binascii.unhexlify (nacl)
1577 defs = ENCRYPTION_PARAMETERS.get(paramversion, None)
1578 version = PDTCRYPT_DEFAULT_VER
1580 kdfname, params = defs ["kdf"]
1582 kdf = kdf_by_version (None, defs)
1583 hsh, _void = kdf (pw, nacl)
1587 if fmt == PDTCRYPT_SCRYPT_INTRANATOR:
1588 out = json.dumps ({ "salt" : base64.b64encode (nacl).decode ()
1589 , "key" : base64.b64encode (hsh) .decode ()
1590 , "paramversion" : paramversion })
1591 elif fmt == PDTCRYPT_SCRYPT_PARAMETERS:
1592 out = json.dumps ({ "salt" : binascii.hexlify (nacl).decode ()
1593 , "key" : binascii.hexlify (hsh) .decode ()
1594 , "version" : version
1595 , "scrypt_params" : { "N" : params ["N"]
1596 , "r" : params ["r"]
1597 , "p" : params ["p"]
1598 , "dkLen" : params ["dkLen"] } })
1600 raise RuntimeError ("bad scrypt output scheme %r" % fmt)
1605 def mode_scan (pw, fname, nacl=None):
1607 Dissect a binary file, looking for PDTCRYPT headers and objects.
1610 fd = os.open (fname, os.O_RDONLY)
1611 except FileNotFoundError:
1612 noise ("PDT: failed to open %s readonly" % fname)
1617 if PDTCRYPT_VERBOSE is True:
1618 noise ("PDT: scan for potential sync points")
1619 cands = locate_hdr_candidates (fd)
1620 if len (cands) == 0:
1621 noise ("PDT: scan complete: input does not contain potential PDT "
1622 "headers; giving up.")
1624 if PDTCRYPT_VERBOSE is True:
1625 noise ("PDT: scan complete: found %d candidates" % len (cands))
1630 def usage (err=False):
1634 indent = ' ' * len (SELF)
1635 out ("usage: %s SUBCOMMAND { --help" % SELF)
1636 out (" %s | [ -v ] { -p PASSWORD | -k KEY }" % indent)
1637 out (" %s [ { -i | --in } { - | SOURCE } ]" % indent)
1638 out (" %s [ { -n | --nacl } { SALT } ]" % indent)
1639 out (" %s [ { -o | --out } { - | DESTINATION } ]" % indent)
1640 out (" %s [ -D | --no-decrypt ] [ -S | --split ]" % indent)
1641 out (" %s [ -f | --format ]" % indent)
1644 out ("\t\tSUBCOMMAND main mode: { process | scrypt }")
1646 out ("\t\t process: extract objects from PDT archive")
1647 out ("\t\t scrypt: calculate hash from password and first object")
1648 out ("\t\t-p PASSWORD password to derive the encryption key from")
1649 out ("\t\t-k KEY encryption key as 16 bytes in hexadecimal notation")
1650 out ("\t\t-s enforce strict handling of initialization vectors")
1651 out ("\t\t-i SOURCE file name to read from")
1652 out ("\t\t-o DESTINATION file to write output to")
1653 out ("\t\t-n SALT provide salt for scrypt mode in hex encoding")
1654 out ("\t\t-v print extra info")
1655 out ("\t\t-S split into files at object boundaries; this")
1656 out ("\t\t requires DESTINATION to refer to directory")
1657 out ("\t\t-D PDT header and ciphertext passthrough")
1658 out ("\t\t-f format of SCRYPT hash output (“default” or “parameters”)")
1660 out ("\tinstead of filenames, “-” may used to specify stdin / stdout")
1662 sys.exit ((err is True) and 42 or 0)
1672 def parse_argv (argv):
1674 mode = PDTCRYPT_DECRYPT
1679 scrypt_format = PDTCRYPT_SCRYPT_DEFAULT
1682 SELF = os.path.basename (next (argvi))
1685 rawsubcmd = next (argvi)
1686 subcommand = PDTCRYPT_SUB [rawsubcmd]
1687 except StopIteration:
1688 bail ("ERROR: subcommand required")
1690 bail ("ERROR: invalid subcommand “%s” specified" % rawsubcmd)
1696 except StopIteration:
1697 bail ("ERROR: argument list incomplete")
1699 def checked_secret (t, arg):
1704 bail ("ERROR: encountered “%s” but secret already given" % arg)
1707 if arg in [ "-h", "--help" ]:
1710 elif arg in [ "-v", "--verbose", "--wtf" ]:
1711 global PDTCRYPT_VERBOSE
1712 PDTCRYPT_VERBOSE = True
1713 elif arg in [ "-i", "--in", "--source" ]:
1714 insspec = checked_arg ()
1715 if PDTCRYPT_VERBOSE is True: noise ("PDT: decrypt from %s" % insspec)
1716 elif arg in [ "-p", "--password" ]:
1717 arg = checked_arg ()
1718 checked_secret (PDTCRYPT_SECRET_PW, arg)
1719 if PDTCRYPT_VERBOSE is True: noise ("PDT: decrypting with password")
1721 if subcommand == PDTCRYPT_SUB_PROCESS:
1722 if arg in [ "-s", "--strict-ivs" ]:
1723 global PDTCRYPT_STRICTIVS
1724 PDTCRYPT_STRICTIVS = True
1725 elif arg in [ "-o", "--out", "--dest", "--sink" ]:
1726 outsspec = checked_arg ()
1727 if PDTCRYPT_VERBOSE is True: noise ("PDT: decrypt to %s" % outsspec)
1728 elif arg in [ "-f", "--force" ]:
1729 global PDTCRYPT_OVERWRITE
1730 PDTCRYPT_OVERWRITE = True
1731 if PDTCRYPT_VERBOSE is True: noise ("PDT: overwrite existing files")
1732 elif arg in [ "-S", "--split" ]:
1733 mode |= PDTCRYPT_SPLIT
1734 if PDTCRYPT_VERBOSE is True: noise ("PDT: split files")
1735 elif arg in [ "-D", "--no-decrypt" ]:
1736 mode &= ~PDTCRYPT_DECRYPT
1737 if PDTCRYPT_VERBOSE is True: noise ("PDT: not decrypting")
1738 elif arg in [ "-k", "--key" ]:
1739 arg = checked_arg ()
1740 checked_secret (PDTCRYPT_SECRET_KEY, arg)
1741 if PDTCRYPT_VERBOSE is True: noise ("PDT: decrypting with key")
1743 bail ("ERROR: unexpected positional argument “%s”" % arg)
1744 elif subcommand == PDTCRYPT_SUB_SCRYPT:
1745 if arg in [ "-n", "--nacl", "--salt" ]:
1746 nacl = checked_arg ()
1747 if PDTCRYPT_VERBOSE is True: noise ("PDT: salt key with %s" % nacl)
1748 elif arg in [ "-f", "--format" ]:
1749 arg = checked_arg ()
1751 scrypt_format = PDTCRYPT_SCRYPT_FORMAT [arg]
1753 bail ("ERROR: invalid scrypt output format %s" % arg)
1754 if PDTCRYPT_VERBOSE is True:
1755 noise ("PDT: scrypt output format “%s”" % scrypt_format)
1757 bail ("ERROR: unexpected positional argument “%s”" % arg)
1758 elif subcommand == PDTCRYPT_SUB_SCAN:
1762 if PDTCRYPT_VERBOSE is True:
1763 noise ("ERROR: no password or key specified, trying $PDTCRYPT_PASSWORD")
1764 epw = os.getenv ("PDTCRYPT_PASSWORD")
1766 checked_secret (PDTCRYPT_SECRET_PW, epw.strip ())
1769 if PDTCRYPT_VERBOSE is True:
1770 noise ("ERROR: no password or key specified, trying $PDTCRYPT_KEY")
1771 ek = os.getenv ("PDTCRYPT_KEY")
1773 checked_secret (PDTCRYPT_SECRET_KEY, ek.strip ())
1776 if subcommand == PDTCRYPT_SUB_SCRYPT:
1777 bail ("ERROR: scrypt hash mode requested but no password given")
1778 elif mode & PDTCRYPT_DECRYPT:
1779 bail ("ERROR: encryption requested but no password given")
1781 if subcommand == PDTCRYPT_SUB_SCAN:
1783 bail ("ERROR: please supply an input file for scanning")
1785 bail ("ERROR: input must be seekable; please specify a file")
1786 return True, partial (mode_scan, secret [1].encode (), insspec, nacl)
1788 if subcommand == PDTCRYPT_SUB_SCRYPT:
1789 if secret [0] == PDTCRYPT_SECRET_KEY:
1790 bail ("ERROR: scrypt mode requires a password")
1791 if insspec is not None and nacl is not None \
1792 or insspec is None and nacl is None :
1793 bail ("ERROR: please supply either an input file or "
1798 if insspec is not None or subcommand != PDTCRYPT_SUB_SCRYPT:
1799 ins = deptdcrypt_mk_stream (PDTCRYPT_SOURCE, insspec or "-")
1801 if subcommand == PDTCRYPT_SUB_SCRYPT:
1802 return True, partial (mode_scrypt, secret [1].encode (), ins, nacl,
1805 if mode & PDTCRYPT_SPLIT: # destination must be directory
1806 if outsspec is None or outsspec == "-":
1807 bail ("ERROR: split mode is incompatible with stdout sink")
1811 os.makedirs (outsspec, 0o700)
1812 except FileExistsError:
1813 # if it’s a directory with appropriate perms, everything is
1814 # good; otherwise, below invocation of open(2) will fail
1816 outs = os.open (outsspec, os.O_DIRECTORY, 0o600)
1817 except FileNotFoundError as exn:
1818 bail ("ERROR: cannot create target directory “%s”" % outsspec)
1819 except NotADirectoryError as exn:
1820 bail ("ERROR: target path “%s” is not a directory" % outsspec)
1823 outs = deptdcrypt_mk_stream (PDTCRYPT_SINK, outsspec or "-")
1825 return True, partial (mode_depdtcrypt, mode, secret, ins, outs)
1829 ok, runner = parse_argv (argv)
1831 if ok is True: return runner ()
1836 if __name__ == "__main__":
1837 sys.exit (main (sys.argv))