package sbaz.manual object Manual { import Documents._ val intro = Section("Overview", "Scala Bazaars supports Scala enthusiasts in sharing the software "& "they create with each other. The community is strengthened when "& "people can help each other out by sharing code. A programming "& "language needs an archive of library code to be its most useful.", "Scala Bazaars has several features specialized for this kind of "& "community:", BulletList( "Since programming efforts in the community are distributed "& "around the world, the system must allow loose, decoupled "& "collaboration. Participants are not required to wait for each "& "other nor for any centralized authority. ", "Since subgroups of the community have their own "& "needs"&MDash&"just consider groups doing commercial "& "development!"&"the system should support subgroup-specific "& "access control policies.", "Since many open source projects are updated frequently, the "& "system must allow conveniently updating components as new "& "versions become available.", "Since, even in the common open-source group, there are "& "different tolerances for stable versus new offerings, the system "& "supports having multiple public servers with different update "& "policies"), "Scala Bazaars is most closely related to Debian's APT, though "& "it shares spirit with systems including: YUM, FreeBSD ports, "& "CPAN, CTAN, SqueakMap, and Fink. The main difference from APT "& "is that Bazaars tries to make it not only possible, but "& "convenient, for subgroups to run their own servers with their "& "own access policies.", "The present document is a reference manual for Scala Bazaars. "& "It is complete at the expense of readable. It does include full "& "specifications for all components of the system, including the "& "command-line interface, all file formats, and the network "& "protocol. Those wanting to use the system should first check "& "whether a tutorial exists for the specific need. Those wanting "& "a more philosophical look at the system, including comparisons "& "to other systems and to the literature, should look at the "& "Package Universes Architecture document. It is available at the "& "Scala Bazaars home page:"& CodeSample("http://lamp.epfl.ch/~spoon/sbaz/")) val concepts = Section("Architectural Concepts", "This section describes the architecture as a whole. Each "& "component of the architecture is described in detail in a later "& "section.", "Users of the system have a " & Emph("local managed "& "directory") &" that includes content supplied by the system. "& " In general, users should not modify content in a managed directory "& "except via a Scala Bazaars system tool. The one exception is the "& Mono("config")& " directory, which is explicitly "& "intended for user modification and should not be modified by the "& "Scala Bazaars tool.", "Sharable content is held by "&Emph("packages")&". "& "A package can be installed into a local managed directory, in "& "which case all of its included files are extracted into the "& "managed directory. A package can also be removed from a local "& "managed directory, causing all of the files it installed to be "& "removed.", "Each package has a name and a version. Packages with the "& "same name may be substituted for each other, but packages with "& "larger version numbers are preferable in some sense. ", "A "&Emph("bazaar")&" is an evolving set of "& "packages. Each managed directory is associated with one bazaar. "& "The system makes it convenient to choose and install packages from "& "a managed directory's own bazaar, but not from other bazaars. The "& "set of available packages evolves as the community associated with "& "the bazaar shares and retracts packages over time. ", "Packages may depend on other packages. All dependencies are "& "resolved within a single bazaar. The system does not allow "& "installing or removing packages in a way that a managed "& "directory's dependencies become unsatisfied. ", "On the whole, this approach gives convenient upgrades "& "(simply grab the newest version of everything), loose coupling "& "(you can wait for a while to upgrade), and convenient dependency "& "management (requesting the install of a package automatically "& "installs the packages it depends on). The cost of this approach "& "is that packages must be posted separately to each bazaar they are "& "useful in, but as argued in the architecture document, that cost "& "appears to be ephemeral. ", "Bazaars may be combined in a variety of ways. Each bazaar "& "has its own access policy, and thus combinations of bazaars with "& "different access policies can yield a wide variety of useful "& "configurations. ", "Bazaars do not hold the packages themselves. Instead, a "& "bazaar holds package advertisements, and the packages themselves "& "must be posted separately. Each package advertisement includes a "& "URL referencing the associated package. ") val bazaarDefns = Section("Bazaar Definitions", "There are a number of kinds of bazaars. They are each "& "described here along with the XML used to formally describe each "& "kind. ", Section("Simple Bazaar", "A simple bazaar includes the packages advertised on a single "& "bazaar server. An example XML description of a simple bazaar is: "& CodeSample( "\n"+ " scala-dev\n" + " http://scala-webapps.epfl.ch/sbaz/scala-dev\n"+ ""), "The location tag specifies which Bazaars server supplies "& "packages for this universe. The name is used by the command-line "& "interface to Scala Bazaars whenever it is necessary to specify a "& "specific simple bazaar within a compound bazaar. "), Section("Empty Bazaar", "The empty bazaar holds no packages at all. Its XML description "& "is simply: "& CodeSample("")), Section("Override Bazaar", "An override bazaar combines the packages from multiple other "& "bazaars, with packages in later bazaars overriding packages in "& "earlier ones. The overriding is name-based: a package in a later "& "bazaar causes "&Emph("any")&" same-named package in an "& "earlier bazaar to disappear from the combined override bazaar. ", "An example XML description is as follows: "& CodeSample( "\n"+ "\n"+ " \n"+ " scala-dev\n"+ " http://scbaztmp.lexspoon.org:8006/scala-dev\n"+ " \n"+ " \n"+ " local-hacks\n"+ " http://localhost/sbaz/local-hacks\n"+ " \n"+ "\n"+ "\n")), Section("Filter Bazaar (not yet implemented)", "A filter bazaar includes those packages from some other "& "bazaar whose name matches a designated regular expression. Filter "& "bazaars are thus useful for taking just a few packages from an "& "existing server. "), Section("Literal Bazaar (not yet implemented)", "A literal bazaar includes a fixed set of packages. Literal "& "bazaars are mainly useful for testing.")) val accessControl = Section("Access Control", "Many organizations that use Scala Bazaars do not want to "& "allow arbitrary accesses from unknown users. Scala Bazaars "& "includes a simple access control system to limit usage as "& "appropriate for the organization. ", "The approach is based on "&Emph("keys")&". A "& "bazaar server has a list of keys that it considers valid. Each "& "request to a server must either include a valid and sufficient "& "key, or it must be in the subset of requests configured as "& Emph("open access")&" for the particular server. ", "Each key is associated with a "&Emph("message pattern")&" "& "denoting the subset of requests it can authorize. The following "& "message patterns are supported: ", TitledPara("Read", "Requests that ask for the "& "list of available packages recorded on a bazaar server. In XML, "& "it looks like:"& CodeSample("")), TitledPara("Edit", "Requests that modify the list "& "of available packages recorded on the server, including posting "& "new advertisements, removing advertisements, and updating an "& "advertisement in place. Edit request patterns include a regular "& "expression that limits the request pattern to those requests "& "involving a package whose name matches the regular expression. "& "The semantics of the supplied regular expression are as for Java's "& "regular-expression library. In XML, it looks "& "like:"& CodeSample("")), TitledPara("Key Edit", "Requests that modify the "& "list of valid keys known to a server. There is only one "& "request-pattern for key editing. Any client that knows a valid "& "key-edit key can perform arbitrary manipulations on the set of keys "& "known to a server (and thus transitively has full access to "& "the server). In XML, a key edit pattern looks like:"& CodeSample("")), "A key includes three pieces of information: a request pattern, a "& "short, descriptive text, and a string of decimal digits. The "& "descriptive text is human-readable and allows keys to be "& "associated with principles in an external access control system. "& "Typical uses are names, email addresses, usernames, and employee "& "ID's. The string of decimal digits is typically randomly "& "generated and must be unguessable in order for the server to "& "remain protected. ", "In XML, a key looks like: "& CodeSample( "\n"+ " \n"+ " \n"+ " \n"+ " lex@lexspoon.org\n"+ " 26405971450520721508638067086623258803\n"+ ""), "The keys-based access control system of Scala Bazaars is both "& "usable directly and supports, it is hoped, a variety of useful "& "access policies. For simple access policies, the server "& "administrator (or someone to whom they delegate) can manually "& "manage and distribute keys. Note that key data can be transmitted "& "via email and web sites, thus allowing distribution of keys to "& "piggyback on existing secured infrastructure. ", "For organizations that have too many users for manual management "& "of access, the keys approach supports writing a simple web server "& "that can manage and distribute keys in an arbitrary fashion. For "& "example, one could write a web server that gives out keys to "& "people who have a valid login according to an LDAP server; "& "nightly, the server could revoke keys for any users whose LDAP "& "entry has disappeared. ") val commonConfigs = Section("Common Configurations", "The available bazaar types and access-control policies "& "described above can be combined to form a number of useful "& "configurations. This section outlines several of them. ", TitledPara("No restrictions", "Wikis have proven "& "that effective and useful communities can be built even with no "& "security restrictions at all. This policy can be implemented by "& "configuring the server with all requests as open "& "access."), TitledPara("Full access, but only to community members", "Many open-source projects have an "& "organization of this form, including Debian's "& Quote("Debian Developers")&" "& " and FreeBSD's "&Quote("committers.")&" This "& "policy can be implemented by giving a key-editing key to the "& "membership gatekeepers. The last step of admitting a new member "& "to the community is to give them their own keys to "& "manipulate the community bazaar servers. "), TitledPara("Single provider, multiple users", "An individual developer wants to provide a suite of packages "& "to be used by a larger community"&MDash&"possibly the world, or "& "possibly a limited base of subscribers. To implement this policy, "& "the developer can keep editing keys for personal use but publish "& "the read key to the desired user "& "base. "), TitledPara("Moderation queues", "Sometimes it is "& "desirable to have a large community contribute suggested packages, "& "but a smaller group to decide on what is actually included. A "& "moderation queue can be implemented as a separate bazaar where a "& "full edit key (with regular expression "&Quote(".*")&" "& "is made available to whichever community is allowed to contribute "& "suggestions (perhaps the entire world). Moderators would have "& "both a read key for the moderation queue and a full edit "& "key for the main bazaar, and could copy packages from the "& "queue to the main bazaar whenever they are deemed "& "appropriate. "), TitledPara("Private local development", "Individual groups can form their own bazaar for private "& "development without needing to coordinate with any central "& "organization. They simply create the bazaar and share keys with "& "members of the group according to their own local security "& "policy. "), TitledPara("Public libraries plus local development", "The above organization can be refined by allowing developers "& "to use publicly available packages even as they develop packages "& "for private use. This policy can be implemented with an override "& "bazaar combining the local bazaar with one or more public "& "bazaars. "), TitledPara("Localized versions of packages", "Local groups may want to use something similar to a "& "widely scoped bazaar, but override specific packages with "& "localized versions. For example, they might prefer a different "& "default language settings of the packages than is used on "& "the widely scoped bazaar. Users want to use the "& "localized version of a package whenever one is available, but the "& "global version otherwise. This policy can be implemented by creating "& "a bazaar server holding the localized packages, and then having "& "users operate in an override bazaar combining this localized "& "bazaar with the public one. "), TitledPara("Stable versus unstable streams", "Projects frequently distinguish between stable and unstable "& "streams of development. The stable streams include packages that "& "are heavily tested and deemed to be reliable, while the unstable "& "streams include packages that are more current and featureful but "& "are not as reliable. A project can implement this policy by "& "having separate bazaar servers for each development stream. Users "& "point their managed directories to the bazaar server appropriate "& "to their needs."), TitledPara("Freezing new stable distribution", "A common process for generating stable distributions of code "& "is to take an unstable stream, start testing it, and disallow any "& "patches except for bug fixes. After some point, the distribution "& "is "&Emph("frozen")&" and considered a stable release. "& "Such processes can be implemented by manipulating the outstanding "& "keys as time passes. The testing phase can be implemented "& "by revoking all outstanding edit keys and switching "& "to a moderation queue process. The final, complete freeze can be "& "implemented by destroying the moderation queue and revoking the "& "remaining edit keys. The frozen bazaar can then "& "be duplicated, with one fork hosting a new development stream "& "while the other becomes is designated a stable "& "release. ")) val packages = Section("Packages and Package Advertisements", Section("Overview", "A Scala Bazaars package is stored in a file with file-ending "& Mono(".sbp")&" (Scala Bazaars Package). The file is in "& "zip format. It should include all of the files that are to be "& "extracted into a managed directory when the package is installed. ", "The "&Mono("meta/")&" directory within a "&Mono(".sbp")&" file "& "does not include files to be extracted, but instead contains "& "meta-information about the package itself. The only defined "& "element of that directory is "& Mono("meta/description")&", which should include an XML "& "document describing the package's contents. All other names "& "within the "&Mono("meta")&" subdirectory are reserved for "& "future use. "), Section("Package Description Files", "An example description file is as follows: "& CodeSample( "\n"+ " sbaz\n"+ " 1.10\n"+ " \n"+ " The command-line interface to Scala Bazaars. Scala\n"+ "Bazaars let you share Scala packages and other goodies with other\n"+ "Scala users.\n"+ ""), "The elements of this description are hopefully "& "self-explanatory. This package is named "&Quote("sbaz")&", its "& "version is 1.10, it has no dependencies, and it has the given "& "human-readable description. "), Section("Package Advertisements", "A package advertisement is simply a package description plus "& "a URL where the package file can be downloaded from. An example "& "of the XML format for a package advertisement is: "& CodeSample( "\n"+ " \n"+ " sbaz\n"+ " 1.10\n"+ " \n"+ " The command-line interface to Scala Bazaars. Scala \n"+ "Bazaars let you share Scala packages and other goodies with other \n"+ "Scala users.\n"+ " \n"+ " http://lamp.epfl.ch/~spoon/scbaztmp/sbaz-1.10.sbp\n"+ ""))) val dependencies = Section("Versions and Dependencies", "Every package has a version. A version is a string that "& "may contain ASCII numbers, digits, and the following symbolic "& "characters: "& CodeSample(".-+/?,&!@#$%^&*"), "Versions are totally ordered by the following algorithm. To "& "compare two versions, first break them into a number of "& "subsequences, each maximally long, where each subsequence only "& "contains decimal digits, only contains alphabetic characters, or "& "only contains symbolic characters. Compare the two sequences "& "lexicographically. Two numeric subsequences are compared numerically, "& "two alphabetic subsequences are compared in ASCII order, and two "& "symbolic subsequences are compared in ASCII order as well. For two "& "subsequences with different kinds of characters, the order is: "& "alphabetic, then numeric, then symbolic. For some examples, the "& "following versions are sorted in order: ", NumberedList( "(empty string)", "abc", "abcd", "abd", "1", "1.1", "1.1a", "1.1a2", "1.1a100", "1.1.5", "1.2", "1.2.", "2", "12"), "A package dependencies can currently only have hard dependencies "& "to specific other package names. For those dependencies to be "& "satisfied, a package must be installed for each of the package names "& "specified. Here is package description for a package that "& "depends on package "&Quote("scala2-library")&" and "& Quote("sbaz")& CodeSample( "\n"+ " \n"+ " base\n"+ " 1.7\n"+ " \n"+ " scala2-library\n"+ " sbaz\n"+ " \n"+ " This package depends on the basic packages that all \n"+ "managed directories must include. Each of these packages is either \n"+ "essential or is very commonly used. \n"+ " \n"+ " http://lamp.epfl.ch/~spoon/scbaztmp/base-1.7.sbp\n"+ ""), "A somewhat richer set of dependencies is planned, including "& "provides, suggests, alternative dependencies, and minimum version "& "numbers. ") val commandLine = Section("Command-Line Interface", Section("Overview", "The command-line interface to Scala Bazaars is the "& Mono("sbaz")&" tool. It is run as follows: "& BlockQuote( "sbaz [-n] [-d "&Emph("dir")&"] "& Emph("command")&" "&Emph("arguments...")), "It always operates on one managed directory. That directory "& "can be specified explicitly with the -d option. If it is not "& "specified, and the "&Mono("sbaz")&" binary is located "& "within a managed directory, then "&Mono("sbaz")&" operates "& "on the managed directory the binary is in. If neither of these "& "cases hold, then "&Mono("sbaz")&" will as a last resort "& "operate on the current directory. In all cases, the tool checks "& "whether the specified directory appears in fact to be a managed "& "directory; if it does not, then it aborts. ", "If -n is specified, then the tool does not perform any work. "& "Instead, it only prints out what it would have done without the "& "-n option. ", "If no "&Emph("command")&" is specified, then "& "the tool prints out a help message. Otherwise, it runs "& "the specified command with the specified arguments. "), Section("Command Reference", "This section will eventually include the entire command "& "reference of the command-line tool. It does not yet. Refer "& "to "&Quote(Mono("sbaz help"))&" to see the "& "tool's built-in documentation. ")) val dirLayout = Section("Suggested Directory Layout", "Each sbaz repository has its own informal standards for the "& "directory layout within a managed directory. This section "& "documents the emerging layout used in the main Scala bazaar. "& "It is the standard for that repository, and it might serve "& "as a guideline for other repositories. ", BulletList( Mono("lib")&MDash&" Any jar file(s) "& "associated with the package, especially those that are meant as "& "libraries to be usable by other programs in the bazaar. Jars "& "placed in this directory are particularly easy to access, because "& "both the generic "&Mono("scala")&" script and most of the "& "tool-running scripts in "&Mono("bin")&" will "& "automatically load classes from any jars in "& ""&Mono("lib")&". Normally, jar filenames in this "& "directory do not include any version number. A typical filename "& "is "&Mono("sbaz.jar")&".", Mono("src")&" "&MDash&" Source code for the "& "package. This source code should be presented in a way that IDE's "& "can find the code easily. Thus far, packages install directories "& "under "&Mono("src")&" that parallel the dotted package "& "paths from the Scala code. For example, class "& Mono("sbaz.clui.CommandLine")&" is found in a file "& "named "& ""&Mono("src/sbaz/clui/CommandLine.scala")&".", Mono("bin")&" "&MDash&" Command-line runnable "& "scripts. These are most easily created via the Scala ant tasks. "& "As a special case, the "&Mono("sbaz")&" tool will make "& "files within "&Mono("bin")&" be executable on platforms "& "where that makes sense.", Mono("config")&" "&MDash&" Configuration "& "files. Packages should not include any files in this directory! "& "They should look in this directory for optional user "& "configuration. If there is a single configuration file, it can be "& "included directly in the "&Mono("config")&" directory, "& "e.g. with a name like "&Mono("config/sbaz.properties")&". "& "If there is more than one configuration file for a package, then "& "the files should be located in a subdirectory of "& Mono("config")&" named after the package name. For "& "example, the "&Mono("sbaz")&" package could include its "& "configuration files in a directory named "& Mono("config/sbaz/")&".", Mono("misc")&" "&MDash&" Arbitrary files not "& "included in any of the above. All such files for a package should "& "be included in a directory named after the package. For example, "& "the "&Mono("sbaz")&" package includes miscellaneous files "& "in the directory "&Mono("misc/sbaz/")&".")) val scalaHome = Section("The scala.home property", "By convention, the "&Mono("scala.home")&" system property "& "is used to point at the root of the current managed directory. "& "Command-line scripts installed in "&Mono("bin")& "directories should normally set this variable. The conventional "& "choice is to set "&Mono("scala.home")&" to the "& Mono("SCALA_HOME")&" environment variable if it set, or "& "otherwise to the parent directory of the "&Mono("bin")& "directory the script is located in. ", "Programs intended to run within a sbaz directory can use "& "this property to locate any files they may need. For example, "& Mono("sbaz")&" itself uses the following code to find "& "a user-specified settings file: "& CodeSample( "val home = System.getProperty(\"scala.home\", \".\") \n"+ "val propFile = new File(new File(new File(home),\n"+ " \"config\"), \"sbaz.properties\")")) val managedDirSpec = Section("Managed Directory Layout", "Most of a managed directory is populated by the contents "& "of packages. The single directory "&Mono("meta")& "is reserved for the system to track information about "& "the managed directory. The "&Mono("meta")&" directory "& "has the following contents: ", BulletList( Mono("universe")&" "&MDash&" a file holding "& "the XML description of the bazaar.", Mono("available")&" "&MDash&" a list of "& "package advertisements available within the bazaar. The file is an "& "XML document whose top level node is "& Mono("")&" and whose subnodes are "& "package advertisements.", Mono("cached")&" "&MDash&" a subdirectory "& "holding a cache of package files that the tool has "& "downloaded.", Mono("installed")&" "&MDash&" information "& "about installed files. The format is described "& "below."), "The "&Mono("installed")&" file holds information "& "about the packages that are currently installed in the managed "& "directory. Its format is mostly straightforward: "& CodeSample( "\n"+ " \n"+ " \n"+ " base\n"+ " 1.7\n"+ " \n"+ " sbaz\n"+ " scala2-library\n"+ " \n"+ " A package that depends on\n"+ "the basic, necessary packages\n"+ " \n"+ " \n"+ " \n"+ " lib\n"+ " \n"+ " \n"+ " \n"+ "\n"+ " \n"+ " \n"+ " sbaz\n"+ " 1.10\n"+ " \n"+ " The command-line interface to Scala Bazaars.\n"+ " \n"+ " \n"+ " \n"+ " bin \n"+ " sbaz\n"+ " \n"+ "\n"+ " \n"+ " bin\n"+ " sbaz.bat\n"+ " \n"+ "\n"+ " \n"+ " misc \n"+ " sbaz\n"+ " scala2-library.jar\n"+ " \n"+ "\n"+ " \n"+ " misc \n"+ " sbaz \n"+ " smartrun.mswin.template\n"+ " \n"+ "\n"+ " \n"+ " misc \n"+ " sbaz \n"+ " smartrun.unix.template\n"+ " \n"+ "\n"+ " \n"+ " lib \n"+ " sbaz.jar\n"+ " \n"+ " \n"+ " \n"+ ""), "Notice that an installed package is the combination of "& "a package plus a list of files. Each file in the list of files "& "is represented with a "&Mono("filename")&" whose "& "attributes determine whether it is an absolute file (never) "& "and whether it is a file (versus a directory). The contents "& "of the "&Mono("filename")&" are the sequence of path "& "components of the file, relative to the managed directory's root. ") val commonProblems = Section("Common Problems", Section("Firewalls and HTTP Proxies", "Scala Bazaars uses HTTP to communicate with universe "& "servers. If your network blocks HTTP access, then, you need to "& "configure sbaz to use an HTTP proxy. To do this, create a file "& "named "&Mono("config/sbaz.properties")&" in your "& "managed directory and give it the appropriate proxy settings, "& "something like: "& CodeSample( "http.proxySet=true\n"+ "http.proxyHost=localhost\n"+ "http.proxyPort=3128\n"))) def wholeThing = new Document { title = "Scala Bazaars Manual" date = "March 2006" author = "Alexander Spoon" sections = List( intro, concepts, bazaarDefns, accessControl, commonConfigs, packages, dependencies, commandLine, dirLayout, scalaHome, managedDirSpec, commonProblems) } }