Changeset 25527 in project
- Timestamp:
- 11/20/11 18:11:42 (9 years ago)
- Location:
- release/4/ugarit/trunk
- Files:
-
- 6 edited
Legend:
- Unmodified
- Added
- Removed
-
release/4/ugarit/trunk/README.txt
r25525 r25527 797 797 hash collisions. Maybe have levels of double-check-ness. 798 798 799 * Everywhere I use (sql ...) to create an sqlite prepared statement,800 don't. Create them all up-front and reuse the resulting statement801 objects, it'll save memory and time. (done for backend-fs/splitlog802 and backend/cache, file-cache still needs it).803 804 799 * Migrate the source repo to Fossil (when there's a 805 800 kitten-technologies.co.uk migration to Fossil), and update the egg 806 801 locations thingy. 807 802 803 * Profile the system. As of 1.0.1, having done the periodic SQLite 804 commits improvement, Ugarit is doing around 250KiB/sec on my home 805 fileserver, but using 87% CPU in the ugarit procesa and 25% in the 806 backend-fs process, when dealing with large files (so full 1MiB 807 blocks are being processed). This suggests that the main 808 block-handling loop in `store-file!` is less than efficient; reading 809 via `current-input-port` rather than using the POSIX egg `file-read` 810 functions may be a mistake, and there is probably more copying afoot 811 than we need. 812 808 813 ## Backends 809 810 * Look at http://bugs.call-cc.org/ticket/492 - can this help?811 814 812 815 * Extend the backend protocol with a special "admin" command that … … 833 836 834 837 * Support for flushing the cache on a backend-cache, via an admin 835 command .838 command, rather than having to delete the cache file. 836 839 837 840 * Support for unlinking in backend-splitlog, by marking byte ranges as … … 845 848 interface, along with the option to compact any or all files. 846 849 847 * Have read-only and unlinkable config flags in the backend-split848 metadata file, settable via admin commands.850 * Have read-only and unlinkable and block size config flags in the 851 backend-split metadata file, settable via admin commands. 849 852 850 853 * For people doing remote backups who want to not hog resources, write … … 910 913 ## Core 911 914 915 * SIGINFO support. Add a SIGINFO handler that sets a flag, and make 916 the `store-file!` and `store-directory!` main loops look for the 917 flag and, if set, display what path we're working on, and perhaps a 918 quick summary of the bytes/blocks stored/skipped stats. 919 920 * Look at http://bugs.call-cc.org/ticket/492 - can we now ditch our 921 own POSIX wrappers and use this egg? 922 912 923 * Log all WARNINGs produced during a snapshot job, and attach them to 913 924 the snapshot object as a text file. … … 920 931 921 932 * Clarify what characters are legal in block keys. Ugarit will only 922 issue hex charactersfor normal blocks, but may use other characters923 for special metadata blocks; establish a contract of what backends924 must support (a-z, A-Z, 0-9, hyphen?)933 issue [a-zA-Z0-9] for normal blocks, but may use other characters 934 (hash?) for special metadata blocks; establish a contract of what 935 backends must support (a-z, A-Z, 0-9, hash?) 925 936 926 937 * API documentation for the modules we export … … 977 988 pointer from the snapshot, and write the snapshot handling code to 978 989 expect this. Again, check Box Backup for that. 979 980 * Some kind of accounting for storage usage by snapshot. It'd be nice981 to track, as we write a snapshot to the archive, how many bytes we982 reuse and how many we back up. We can then store this in the983 snapshot metadata, and so report them somewhere. The blocks uploaded984 by a snapshot may well then be reused by other snapshots later on,985 so it wouldn't be a true measure of 'unique storage', nor a measure986 of what you'd reclaim by deleting that snapshot, but it'd be987 interesting anyway.988 990 989 991 * Option, when backing up, to not cross mountpoints … … 1152 1154 1153 1155 * 1.0.2: Made the file cache also commit periodically, rather than on 1154 every write, in order to improve performance. 1156 every write, in order to improve performance. Counting blocks and 1157 bytes uploaded / reused, and file cache bytes as well as hits; 1158 reporting same in snapshot UI and logging same to snapshot metadata. 1155 1159 1156 1160 * 1.0.1: Consistency check on read blocks by default. Removed warning -
release/4/ugarit/trunk/backend-cache.scm
r25521 r25527 2 2 (use sql-de-lite) 3 3 (use matchable) 4 (use miscmacros) 4 5 5 6 (define cache-sql-schema … … 27 28 (set! *updates-since-last-commit* 0)) 28 29 (define (maybe-flush!) 29 (set! *updates-since-last-commit* 30 (+ *updates-since-last-commit* 1)) 30 (inc! *updates-since-last-commit*) 31 31 (when (> *updates-since-last-commit* commit-interval) 32 32 (flush!))) -
release/4/ugarit/trunk/backend-fs.scm
r25521 r25527 4 4 (use matchable) 5 5 (use regex) 6 (use miscmacros) 6 7 7 8 (define (backend-fs base) … … 215 216 (set! *updates-since-last-commit* 0))) 216 217 (maybe-flush! (lambda () 217 (set! *updates-since-last-commit* 218 (+ *updates-since-last-commit* 1)) 218 (inc! *updates-since-last-commit*) 219 219 (when (> *updates-since-last-commit* commit-interval) 220 220 (flush!)))) -
release/4/ugarit/trunk/test/run.scm
r25479 r25527 478 478 (cons snapshot acc)) 479 479 '())) 480 (pp result) 480 481 (test-assert "History has expected form" 481 482 (match result 482 483 (((('previous . sk1*) 483 484 ('mtime . _) 484 ('contents . dir2-key*)) 485 ('contents . dir2-key*) 486 ('stats . _)) 485 487 (('mtime . _) 486 ('contents . dir-key*))) 488 ('contents . dir-key*) 489 ('stats . _))) 487 490 (and (string=? sk1 sk1*) 488 491 (string=? dir2-key dir2-key*) … … 502 505 (test-define-values "Walk the history of tag 'Test' with fold-archive-node" (tag) 503 506 (fold-archive-node a '(tag . "Test") (lambda (name dirent acc) (cons (cons name dirent) acc)) '())) 507 (pp tag) 504 508 (test-assert "Tag history has expected form" 505 509 (match tag … … 509 513 ('previous . sk1*) 510 514 ('mtime . _) 511 ('contents . dir-key*)) 515 ('contents . dir-key*) 516 ('stats . _)) 512 517 (dir-key-c** 513 518 _ … … 515 520 ('previous . sk1**) 516 521 ('mtime . _) 517 ('contents . dir-key**)) 522 ('contents . dir-key**) 523 ('stats . _)) 518 524 (dir-key-c*** 519 525 _ 520 526 'snapshot 521 527 ('mtime . _) 522 ('contents . dir-key***))) 528 ('contents . dir-key***) 529 ('stats . _))) 523 530 (and 524 531 (string=? sk1 sk1*) -
release/4/ugarit/trunk/ugarit-core.scm
r25525 r25527 4 4 archive-hash 5 5 archive-global-directory-rules 6 archive-snapshot-blocks-stored 7 archive-snapshot-bytes-stored 8 archive-snapshot-blocks-skipped 9 archive-snapshot-bytes-skipped 6 10 archive-file-cache-hits 11 archive-file-cache-bytes 7 12 archive-writable? 8 13 archive-unlinkable? … … 105 110 ;; 106 111 ;; THE ARCHIVE 112 ;; This thing is becoming a bit of a God Object. Figure out how to 113 ;; refactor it a bit, perhaps? 107 114 ;; 108 115 … … 118 125 decrypt ; the decryptor, inverse of the above 119 126 global-directory-rules ; top-level directory rules 127 128 ; Snapshot counters 129 (setter snapshot-blocks-stored) ; Blocks written to storage 130 (setter snapshot-bytes-stored) ; Bytes written to storage 131 (setter snapshot-blocks-skipped) ; Blocks already in storage and reused (not including file cache wins) 132 (setter snapshot-bytes-skipped) ; Bytes already in storage and reused (not including file cache wins) 133 134 ; File cache 120 135 file-cache ; sqlite db storing filesystem cache (see store-file! procedure); #f if not enabled 121 136 file-cache-get-query ; sqlite stored procedure 122 137 file-cache-set-query ; sqlite stored procedure 123 file-cache-updates-uncommitted ; count of updates since last commit 124 file-cache-hits ; count of file cache hits 138 (setter file-cache-updates-uncommitted) ; count of updates since last commit 139 (setter file-cache-hits) ; count of file cache hits 140 (setter file-cache-bytes) ; count of file cache bytes saved 125 141 ) 126 142 … … 133 149 (exec (sql (archive-file-cache archive) "commit;")) 134 150 (exec (sql (archive-file-cache archive) "begin;")) 135 ( archive-file-cache-updates-uncommitted-set! archive0))151 (set! (archive-file-cache-updates-uncommitted archive) 0)) 136 152 (exec (archive-file-cache-set-query archive) 137 file-path mtime size key)) 153 file-path mtime size key) 154 (inc! (archive-file-cache-updates-uncommitted archive))) 138 155 139 156 (define (file-cache-get archive file-path mtime size) … … 295 312 decrypt 296 313 *global-rules* 314 ; Snapshot counters 315 0 0 0 0 316 ; File cache 297 317 *file-cache* 298 318 (if *file-cache* (sql *file-cache* "SELECT key FROM files WHERE path = ? AND mtime = ? AND size = ?") #f) 299 319 (if *file-cache* (sql *file-cache* "INSERT OR REPLACE INTO files (path,mtime,size,key) VALUES (?,?,?,?)") #f) 300 0 0 ))))320 0 0 0)))) 301 321 302 322 ; Take a block, and return a compressed and encrypted block … … 327 347 (signal (make-property-condition 'exn 'location 'check-archive-unlinkable 'message "This isn't an unlinkable archive - it's append-only")))) 328 348 349 (define (archive-log-reuse! archive data) 350 (inc! (archive-snapshot-blocks-skipped archive)) 351 (inc! (archive-snapshot-bytes-skipped archive) (u8vector-length data))) 352 329 353 (define (archive-put! archive key data type) 330 ( if(not (archive-writable? archive))354 (when (not (archive-writable? archive)) 331 355 (signal (make-property-condition 'exn 'location 'archive-put! 'message "This isn't a writable archive"))) 332 ((storage-put! (archive-storage archive)) key (wrap-block archive data) type)) 356 ((storage-put! (archive-storage archive)) key (wrap-block archive data) type) 357 (inc! (archive-snapshot-blocks-stored archive)) 358 (inc! (archive-snapshot-bytes-stored archive) (u8vector-length data)) 359 (void)) 333 360 334 361 (define (archive-exists? archive key) … … 426 453 427 454 (if (archive-exists? archive hash) 428 (values (reusing hash) #t) 455 (begin 456 (archive-log-reuse! archive data) 457 (values (reusing hash) #t)) 429 458 (begin 430 459 (archive-put! archive hash data type) … … 477 506 (set! *key-buffer-bytes* 0) 478 507 (set! *key-buffer-reused?* #t) 508 (archive-log-reuse! archive keys-serialised) 479 509 (values (reusing hash) #t)) ; We, too, are reused 480 510 (begin ; We are unique and new and precious! … … 599 629 (if cache-result ;; FIXME: This assumes that the cached file IS in the archive. Give a configurable option to make it check this, making the file-cache a file hash cache rather than also being an archive presence cache like backend-cache as well, for safety. 600 630 (begin 601 ( archive-file-cache-hits-set! archive602 (+ (archive-file-cache-hits archive) 1))631 (inc! (archive-file-cache-hits archive)) 632 (inc! (archive-file-cache-bytes archive) size) 603 633 (values cache-result #t)) ; Found in cache! Woot! 604 634 (store-file-and-cache! mtime size))) ; not in cache … … 666 696 (set! *key-buffer* '()) 667 697 (set! *key-buffer-reused?* #t) 698 (archive-log-reuse! archive serialised-buffer) 668 699 (values (reusing hash) #t)) ; We, too, are reused 669 700 (begin ; We are unique and new and precious! … … 993 1024 994 1025 (if (archive-exists? archive hash) 995 (values (reusing hash) #t) 1026 (begin 1027 (archive-log-reuse! archive data) 1028 (values (reusing hash) #t)) 996 1029 (begin 997 1030 (for-each (lambda (key) … … 1017 1050 ;; 'notes (user-supplied notes) 1018 1051 ;; 'previous (hash of previous snapshot) 1052 ;; 'stats (alist of stats: 1053 ;; 'blocks-stored 1054 ;; 'bytes-stored 1055 ;; 'blocks-skipped 1056 ;; 'bytes-skipped 1057 ;; 'file-cache-hits 1058 ;; 'file-cache-bytes 1019 1059 ;; Returns the snapshot's key. 1020 1060 (define (tag-snapshot! archive tag contents-key contents-reused? snapshot-properties) 1021 1061 (check-archive-writable archive) 1022 1062 (archive-lock-tag! archive tag) 1023 (let ((previous (archive-tag archive tag)) 1024 (snapshot 1025 (append 1026 (list 1027 (cons 'mtime (current-seconds)) 1028 (cons 'contents contents-key)) 1029 snapshot-properties)) 1030 (keys 1031 (list ; We do not list the previous snapshot - since we are about to overwrite the tag that points to it, which would be a decrement. 1032 (cons contents-key contents-reused?)))) 1063 (let* ((previous (archive-tag archive tag)) 1064 (stats (list 1065 (cons 'blocks-stored (archive-snapshot-blocks-stored archive)) 1066 (cons 'bytes-stored (archive-snapshot-bytes-stored archive)) 1067 (cons 'blocks-skipped (archive-snapshot-blocks-skipped archive)) 1068 (cons 'bytes-skipped (archive-snapshot-bytes-skipped archive)) 1069 (cons 'file-cache-hits (archive-file-cache-hits archive)) 1070 (cons 'file-cache-bytes (archive-file-cache-bytes archive)))) 1071 (snapshot 1072 (append 1073 (list 1074 (cons 'mtime (current-seconds)) 1075 (cons 'contents contents-key) 1076 (cons 'stats stats)) 1077 snapshot-properties)) 1078 (keys 1079 (list ; We do not list the previous snapshot - since we are about to overwrite the tag that points to it, which would be a decrement. 1080 (cons contents-key contents-reused?)))) 1033 1081 (if previous 1034 1082 (begin -
release/4/ugarit/trunk/ugarit.scm
r25479 r25527 274 274 (cons 'notes *snapshot-notes*))))) 275 275 (printf "Successfully archived ~A to tag ~A\n" fspath tag) 276 (printf "Snapshot hash: ~A\n" snapshot-key) 277 (printf "Written ~A bytes to the archive in ~A blocks, and reused ~A bytes in ~A blocks (before compression)\n" 278 (archive-snapshot-bytes-stored archive) 279 (archive-snapshot-blocks-stored archive) 280 (archive-snapshot-bytes-skipped archive) 281 (archive-snapshot-blocks-skipped archive)) 276 282 (if (positive? (archive-file-cache-hits archive)) 277 (printf "File cache has saved us ~A file hashings \n"278 (archive-file-cache-hits archive) ))279 (printf "Snapshot hash: ~A\n" snapshot-key)283 (printf "File cache has saved us ~A file hashings / ~A bytes (before compression)\n" 284 (archive-file-cache-hits archive) 285 (archive-file-cache-bytes archive))) 280 286 (archive-close! archive)))) 281 287 282 288 (("explore" confpath) 283 289 (let ((archive (open-archive
Note: See TracChangeset
for help on using the changeset viewer.