Changeset 25528 in project

Timestamp:

11/20/11 19:46:01 (9 years ago)

Author:

Alaric Snell-Pym

Message:

ugarit: Used the parley egg for command-line editing.

Location:

release/4/ugarit/trunk

Files:

• LICENCE.txt (modified) (1 diff)
• README.txt (modified) (10 diffs)
• posixextras.scm (deleted)
• ugarit-core.scm (modified) (2 diffs)
• ugarit.meta (modified) (1 diff)
• ugarit.scm (modified) (2 diffs)

release/4/ugarit/trunk/LICENCE.txt

@@ -1 +1 @@
 Copyright (c) 2008-2009, Warhead.org.uk Ltd
+Copyright (c) 2009-2011, Alaric Snell-Pym
 
 All rights reserved.

release/4/ugarit/trunk/README.txt

@@ -611 +611 @@
 entirely uploaded.
 
+Actually, there is *one* caveat: blocks that were uploaded, but never
+make it into a finished snapshot, will be marked as "referenced" but
+there's no snapshot to delete to un-reference them, so they'll never
+be removed when you delete snapshots. (Not that snapshot deletion is
+implemented yet, mind). If this becomes a problem for people, we could
+write a "garbage collect" tool that regenerates the reference counts
+in an archive, leading to unused blocks (with a zero refcount) being
+unlinked.
+
 ## Should I share a single large archive between all my filesystems?
 
@@ -637 +646 @@
 the archive at any point, or who can read the archive itself at their
 leisure.
+
+Ugarit's splitlog backend creates files with "rw-------" permissions
+out of the box to try and prevent this. This is a pain for people who
+want to share archives between UIDs, but we can add a configuration
+option to override this if that becomes a problem.
 
 ### Reading your data
@@ -846 +860 @@
   stats in the index of how many byte ranges are unused, and how many
   bytes unused, in each file, and report them in the info admin
-  interface, along with the option to compact any or all files.
+  interface, along with the option to compact any or all files. We'll
+  need to store refcounts in the backend metadata (should we log
+  reuses, then, so the metadata can always be reconstructed, or just
+  set them to NULL on a reconstruct); when this is enabled on an
+  existing archive with no refcounts, default them to NULL, and treat
+  a NULL refcount as "infinity".
 
 * Have read-only and unlinkable and block size config flags in the
@@ -888 +907 @@
   this, we might need to log an "unknown" for that block key / archive
   pair, rather than assuming the block is not there, and check it
-  later. Users
-  will be given an admin command to notify the backend of an archive
-  going missing forever, which will cause it to be removed from the
-  cache. Affected blocks should be examined and re-replicated if their
-  replication count is now too low. Another command should be
+  later. Users will be given an admin command to notify the backend of
+  an archive going missing forever, which will cause it to be removed
+  from the cache. Affected blocks should be examined and re-replicated
+  if their replication count is now too low. Another command should be
   available to warn of impending deliberate removal, which will again
   remove the archive from the cluster and re-replicate, the difference
   being that the disappearing archive is usable for re-replicating
   FROM, so this is a safe operation for blocks that are only on that
-  one archive. The individual physical archives
-  that we put replication on top of won't be "valid" archives unless
-  they are 100% replicated, as they'll contain references to blocks
-  that are on other archives. It might be a good idea to mark them as
-  such with a special tag to avoid people trying to restore directly
-  from them. A copy of the replication configuration could be stored
-  under a special tag to mark this fact, and to enable easy finding of
-  the proper replicated archive to work from. There should be a
-  configurable option to snapshot the cache to the archives whenever
-  the replicated archive is closed, too. The command line to the
-  backend, "backend-replicated", should point to an sqlite file for
-  the configuration and cache, and users should use admin commands to
-  add/remove/modify archives in the cluster.
+  one archive. The individual physical archives that we put
+  replication on top of won't be "valid" archives unless they are 100%
+  replicated, as they'll contain references to blocks that are on
+  other archives. It might be a good idea to mark them as such with a
+  special tag to avoid people trying to restore directly from them;
+  the frontend should complain if you attempt to directly use an
+  archive with the special tag in place. A copy of the replication
+  configuration could be stored under a special tag to mark this fact,
+  and to enable easy finding of the proper replicated archive to work
+  from. There should be a configurable option to snapshot the cache to
+  the archives whenever the replicated archive is closed, too. The
+  command line to the backend, "backend-replicated", should point to
+  an sqlite file for the configuration and cache, and users should use
+  admin commands to add/remove/modify archives in the cluster.
 
 ## Core
@@ -917 +936 @@
   flag and, if set, display what path we're working on, and perhaps a
   quick summary of the bytes/blocks stored/skipped stats.
-
-* Look at http://bugs.call-cc.org/ticket/492 - can we now ditch our
-  own POSIX wrappers and use this egg?
 
 * Log all WARNINGs produced during a snapshot job, and attach them to
@@ -985 +1001 @@
   given tag. Or just remove X out of Y snapshots older than a given
   number of days on a given tag. We have the core support for this;
-  just find a snapshot and `unlink-directory!` it, leaving a dangling
-  pointer from the snapshot, and write the snapshot handling code to
-  expect this. Again, check Box Backup for that.
+  just find a snapshot and `unlink-directory!` its contents, leaving a
+  dangling pointer from the snapshot, and write the snapshot handling
+  code to expect this. Again, check Box Backup for that.
 
 * Option, when backing up, to not cross mountpoints
@@ -1032 +1048 @@
   any required top-level snapshot objects if the dump doesn't start
   from a snapshot) and putting it onto a specified tag. The
-  intension is that this format can be used to migrate your stuff
+  intention is that this format can be used to migrate your stuff
   between archives, perhaps to change to a better backend.
 
@@ -1038 +1054 @@
 
 * Better error messages
-
-* Line editing in the "explore" CLI, ideally with tab completion
 
 * API mode: Works something like the backend API, except at the
@@ -1050 +1064 @@
   archive, rather than needing to use explore mode. Also the option to
   extract given just a block key (useful when reading from keys logged
-  manually at snapshot time, or from a backend that has a tag log).
+  manually at snapshot time).
 
 * FUSE/9p support. Mount it as a read-only filesystem :-D Then
@@ -1156 +1170 @@
   every write, in order to improve performance. Counting blocks and
   bytes uploaded / reused, and file cache bytes as well as hits;
-  reporting same in snapshot UI and logging same to snapshot metadata.
+  reporting same in snapshot UI and logging same to snapshot
+  metadata. Switched to the `posix-extras` egg and ditched our own
+  `posixextras.scm` wrappers. Used the `parley` egg in the `ugarit
+  explore` CLI for line editing.
 
 * 1.0.1: Consistency check on read blocks by default. Removed warning

release/4/ugarit/trunk/ugarit-core.scm

@@ -100 +100 @@
 (use miscmacros)
 (use posix)
+(use posix-extras)
 (use crypto-tools)
 (use stty)
@@ -140 +141 @@
   (setter file-cache-bytes)                 ; count of file cache bytes saved
   )
-
-(include "posixextras.scm")
 
 (define file-cache-commit-interval 1000)

release/4/ugarit/trunk/ugarit.meta

@@ -3 +3 @@
  (license "BSD")
  (category data)
- (needs miscmacros sql-de-lite crypto-tools srfi-37 stty matchable autoload regex tiger-hash)
+ (needs miscmacros sql-de-lite crypto-tools srfi-37 stty matchable autoload regex tiger-hash posix-extras parley)
  (optional lzma z3 tiger-hash sha2 aes)
  (author "Alaric Snell-Pym")

release/4/ugarit/trunk/ugarit.scm

@@ -6 +6 @@
 (use matchable)
 (use regex)
+(use parley)
 
 (define (bit? i b)
@@ -134 +135 @@
 ;; To get started, call with '() as directory-key and path
 (define (explore-archive archive directory-key path quit-continuation)
-   (printf "~A> " (apply string-append (map (lambda (element) (string-append "/" element)) path)))
-   (let ((line (read-line)))
+   (let ((line (parley (sprintf "~A> " (apply string-append (map (lambda (element) (string-append "/" element)) path))))))
       (if (eq? line #!eof)
          (quit-continuation (void)))