Feedback and more questions.

diff --git a/docs/TODO.txt b/docs/TODO.txt
index 0a175fe..45ac1fc 100644 (file)
--- a/docs/TODO.txt
+++ b/docs/TODO.txt
@@ -7,26 +7,42 @@ can satisfy our the design requirements.
 
    [How did duplicity solve this?]
 
-Answer:
-
-Effectively GNU tar supports multivolume. Tar multivolume has some limitations,
-for example they cannot be compressed.  To implement support for multivolume
-tars in python tarlib shouldn't be very difficult though, but it would needed
-to study the multivolume format.
-
-Duplicity works creating fixed sized volumes, where the files being archived
-are stored. If a file doesn't fill the current volume, it's split between
-current volume and the next. Duplicity generates an external manifest file that
-specifies what file was splitted from in the end of a volume and the begining of
-the other This is how they seem to keep track of the splitted files and the
-volumes themselves.
+Wadobo:
+    Effectively GNU tar supports multivolume. Tar multivolume has some limitations,
+    for example they cannot be compressed.  To implement support for multivolume
+    tars in python tarlib shouldn't be very difficult though, but it would needed
+    to study the multivolume format.
+
+    Duplicity works creating fixed sized volumes, where the files being archived
+    are stored. If a file doesn't fill the current volume, it's split between
+    current volume and the next. Duplicity generates an external manifest file that
+    specifies what file was splitted from in the end of a volume and the begining of
+    the other This is how they seem to keep track of the splitted files and the
+    volumes themselves.
+
+Intra2net:
+    We could implement multi volume by splitting
+    the compressed tar archive once it has reached
+    the volume size limit and treat it later on like one,
+    big virtual volume. The volumes could be encrypted, too.
+
+    -> No need for a (fragile?) manifest file. I think archiving to tape
+        pretty much works like this with gnu tar.
+
+    In case of an emergency you could just "cat"
+    those files together and start unpacking it.
+
+    In case the first volume of a splitted file is gone,
+    we still have the "emergency recover" tool which
+    can extract the files from a given position / search
+    for the first marker.
 
 2. Design how we are going to split huge files over multiple volumes
    and recognize them later on.
 
    [How did duplicity solve this? There are unit tests for this]
 
-Answer:
+Wadobo:
 
 The method used by duplicity related in the previous question seems to do the
 trick without having to resort to using magic markers and escaping, so I would
@@ -43,9 +59,12 @@ Volume 2:
     EndingPath     "Ultimate\x20Lite\x20x86\x20Faster\x20June\x202012.iso" 861
     Hash SHA1 2299f5afc7f41f66f5bd12061ef68962d3231e34
 
+Intra2net:
+    Answered above: Manifest is probably not needed.
+
 3. How can we restart the gzip compression for every file in the tar archive?
 
-Answer:
+Wadobo:
 
 duplicity has already proposed a new better-than-tar file format, which
 might be an interesting way to start collaborating with them: giving them some
@@ -67,6 +86,11 @@ A compromise solution could be to simply just use a good volume size, and try
 to remove the possibility of a file whose size is less than the volume
 size to be splitted in two volumes. Which can be easily done with duplicity.
 
+Intra2net:
+    We want the compression / encryption on top of tar since
+    we don't want to leak the file names / directory
+    structure in the encryption case.
+
 --
 [1] http://duplicity.nongnu.org/new_format.html
 
@@ -90,7 +114,33 @@ do they think about adding that, if they would be willing to add this feature
 upstream, and in any case, ask them if someone has already tried to do this
 before and if they have any suggestion before we start to try to code it.
 
-5. Collect feedback from Intra2net. If all is fine,
+Intra2net:
+    Please go ahead and ask on the duplicity mailinglist two things:
+    - (like you proposed) What do they think about the diff backup mode?
+
+    - What do they think about restarting the gzip compression / encryption
+      on the stream level at each file "boundary".
+      If they don't like this the our future is already sealed :o)
+
+*** New items to check ***
+- please investigate how the compression can be restarted in tarlib
+  on the global stream level for each file boundary.
+  We need this for our "own" solution and/or for duplicity.
+  Only this gives good data integrity.
+
+- we could use a tar format variant like GNU tar or "pax".
+  Please evaluate the pros / cons of other tar format variants.
+ 
+  pax f.e. seems to store the file owner names etc.
+  IIRC Fedora added xattr support to tar.
+
+- duplicity is GPL. We intend to add the "archive layer"
+  later on to our backup subsystem which means the
+  whole subsystem must become GPL, too. That's something
+  I have not made my mind up, but it would block that road.
+  OTOH we save a bit of development time by using the duplicity basis.
+
+*** Later on ***
+X. Collect feedback from Intra2net. If all is fine,
    design the class layout for all the design requirements
    and have another round of feedback.
-