wiki:Clojure Client Tutorial

Version 72 (modified by kedwar10, 13 years ago) (diff)

--

Clojure Client

Getting Started

The first thing you need before using this client is the Leiningen script. This tool makes managing clojure projects relatively easy by providing a REPL, automatically downloading dependencies, compiling your projects into jar files, and many other useful capabilities. You can download Leiningen and read its tutorial here

If you're having trouble with leiningen, type "lein help" in the shell to see available commands (you can configure leiningen individually for each project, however, so these will differ depending on which directory you are in). Leiningen supports integration with a variety of text editors, but the most Clojure developers prefer to use emacs with the swank/slime mechanism (the setup can be quite involved however). An alternative is to use the vimclojure plugin for vim (this is not as powerful as slime, but has the compensating advantage of working straight out of the box). Follow the link for more details.

Once you have the script in a directory of your choice (and have made it executable), add the following to your .cshrc file (assuming you are using the c shell; else lookup specific instructions for your shell) 

setenv PATH "$PATH":"/path/to/leiningen/sript

You could skip this step but it will less convenient later on for debugging.

While a background in Scheme and Java should be sufficient to immediately start working with the client, an excellent overview of the idiomatic use of Clojure can be found in "Joy of Clojure", by Michael Fogus and Chris Houser. If you need to quickly look up the documentation on any particular function, just type "(doc <function name>)" in the REPL (This will also work for functions that you define provided that you wrote docstrings or added meta-data).

Clojure Resources:

Lisp and Scheme resources:

Some advice for Clojure programming:

  • Always strive for pure functions (as in no side effects). This is critical if you want to avoid refactoring your code later on.
  • For tracing functions, use ​http://richhickey.github.com/clojure-contrib/trace-api.html.
  • :pre and :post conditions can be used to put constraints on functions. Here is an example ​http://blog.fogus.me/2009/12/21/clojures-pre-and-post.
  • Prefer working from the leiningen REPL; avoid compiling and running.
  • If you define a record in a namespace and wish to access it in another, you must explicitly import it with the ":import" keyword in your namespace declaration.
  • Giving names to your anonymous functions will dramatically increase the usefulness of Clojure error reporting (an example: (fn identity [x] x). Doing this will also allow the function to call itself.
  • If you want to ensure tail-call optimization, use the "recur" function instead of using your function name. "recur" also works with loops.
  • "Weird" error messages usually result from forgetting to specify the arguments during a function definition or from mismatched parenthesis.
  • Atoms are your go-to way to manage state. (Don't ignore the other types however, as choosing the inappropriate type can force a refactoring later on.)
  • The function that you pass as an argument to "swap!" (the function that updates an atom) must be pure. This is because the update to the atom is retriable so it may get called more then once.
  • If you must define a type, prefer the simplest option. A decision flowchart: ​http://cemerick.com/2011/07/05/flowchart-for-choosing-the-right-clojure-type-definition-form/.
  • If a future fails silently, try just running your function in a regular thread instead.

Organization of Code

Once you have lieningen set up, cd to the client directory and type "tree" into the shell.

This organization is the default for the leiningen projects created with the "lein new" command.

"docs" contains and html file that provides a sort of annotated presentation of the code. Read this file first to familiarize yourself with the organization of the project. Once you have made your own changes to the client and wish to update the documentation, run "lein marg" in the top level directory. This invokes the marginalia jar in the "lib" directory and generates a new html file.

In the top level directory there is a file called "project.clj". This file tells Leiningen how your project is organized. For details, consult the tutorial, but know that if you wish to add additional Clojure libraries to your project, you will need to specify them here, and run "lein deps" to download them. Specific directions for Leiningen are provided nearly universally for Clojure project, so just look them up as needed. 

(defproject client "1.0.0-SNAPSHOT"
            :description "Quagents client"
            :url "http://www.cs.rochester.edu/trac/quagents/wiki/WikiStart"
            :dependencies [[org.clojure/clojure "1.2.1"]
                           [org.clojure/clojure-contrib "1.2.0"]
                           [org.clojars.automata/rosado.processing "1.1.0"]
                           [overtone/at-at "0.0.1"]
                           [org.clojure/core.logic "0.6.2"]
                           [match "0.1.0-SNAPSHOT"]]
            :dev-dependencies [[lein-marginalia "0.6.0"]]
            :main client.core)

"src" contains all of the .clj files in your project. The organization of the directory must correspond to the namespaces of your project so edit with care (more on this later, but one thing to note is that if you have a namespace with dash in it, "client.my-ns", for example, the actual filename must be "my_ns.clj" and it must be located in the client directory).

"test" contains files that Leiningen will use for testing your functions via the "lein test" command. This functionality is not critical in Clojure since you can debug easily from the REPL, but if you want to batch test functions then you can look into this.

When you create a new project with Leiningen, it will automatically provide a core.clj file for you. If your namespaces are organized into a tree, this is the root. When you run the REPL in the top level directory via the "lein repl" command, the "core" namspace (or whatever namespace in project.clj is specified as ":main") becomes available to you. This is why having leiningen on your path is useful, as you won't always want to load the core namespace. An important detail: namespaces must NOT be cyclic in clojure, so plan accordingly when designing your project.

Client Usage Tutorial

Introduction

The client has several ways of writing messages to the server, each of which has different behaviour with respect to synchronization and reply processing. The user must specify the way these functions operate by passing them data structures and functions which will be called on the data returned from the server. This is analogous to the "reduce" or "foldl" functions found in Clojure and Scheme respectively. If you are unfamiliar with using functions as arguments, then it may be worthwhile to experiment with in the REPL with either "map" or "reduce". For example, 

user=> (map inc [1 2 3 4 5])
(2 3 4 5 6)
user=> (map (fn [e] (* e 2)) [1 2 3 4 5]) ; an anonymous function that doubles its argument
(2 4 6 8 10)
user=> (map #(* % 2) [1 2 3 4 5]) ; same, but with alternative syntax        
(2 4 6 8 10)
user=> (reduce + [1 2 3 4 5])
15
user=> (reduce #(cons %2 %1) (list) [1 2 3 4 5]) ; reverse a list
(5 4 3 2 1)

Another useful concept is that of "closures". Generally these appear in the context of a "let over lambda". For example, 

user=> (let [a (atom 0)]
         (defn counter [] 
           (swap! a inc)))
#'user/counter
user=> (counter)
1
user=> (counter)
2
user=> (counter)
3

In this case, counter "closes" over the a lexical variable "a" and prevents it from going out of scope.

Basic Functionality

cd to the top level directory and type "lein repl". If everything went smoothly you'll see something about no rlwrap (if you are working on one of the lab machines) and the prompt. 

$ lein repl
which: no rlwrap in ... (output truncated)
REPL started; server listening on localhost:2192. 
client.core=>

Note that the "core" namespace has loaded. You can launch ioquake with a specific map using the "run-ioquake" function (located in utilities.clj). Type (doc run-ioquake) to see the documentation. 

client.core=> (doc run-ioquake)
-------------------------
client.utilities/run-ioquake
([level] [path level])
  This function launches ioquake with the specified level,
  if the path to ioquake is hardcoded into this function, 
  type '(run-ioquake <level>)' where <level> is a map in 
  the maps folder of ioquake,  else you must provide the path.
nil

For example, 

client.core=> (run-ioquake "/full/path/to/ioquake3.i386" "sat")
#<UNIXProcess java.lang.UNIXProcess@2a5ab9>

If you already have the path predefined, however, this will also work: 

client.core=> (run-ioquake "sat")
#<UNIXProcess java.lang.UNIXProcess@56c3cf>

You should now see ioquake running in a separate window (if you are in fullscreen mode, navigate to setup and switch this off). Now hit backtick (`) to free the mouse from ioquake. Back in the REPL, type "(doc load-quagent)" 

client.core=> (doc load-quagent)
-------------------------
client.commands/load-quagent
([] [moniker])
  This function will load a quagent into the virtual environment.
  Takes a optional 'moniker' (name is a reserved word in clojure)
  argument that specifies the key that can be used to identify the
  quagent and get information about it.  Otherwise a randomly 
  generated name will be made with the prefix 'quagent'.
nil

Note that if there is only one spawnpoint on the map, you'll want to move forward a bit in order to avoid getting telefragged. 

client.core=> (load-quagent)
:quagent277
client.core=> (load-quagent "Bob")
:Bob

You can check which quagents are currently loaded with the 'get-quagents' function. 

client.core=> (get-quagents)
#{:Bob :quagent277}

You can use these keywords to make the quagents do things in the environment (Note: this functions are subject to change). 

client.core=> (move :Bob 500 0)
[]

Note that control of the terminal does not return until the bot has completed his action.

Defining New Commands

Now say you want to define a function on the fly to send an op to the server, this can be done with either "send-and-forget", "send-and-get", "send-and-get-later", and "send-and-watch". These functions provide a human-readable interface to protocol zero by mapping all of the op codes to keywords. Here is the full list of keyword arguments subject to change).

  • :move-indefinitely --> "mi"
  • :move-for --> "mf"
  • :move-by --> "mb"
  • :move-to --> "mt"
  • :jump-once --> "ju"
  • :rotate --> "ro"
  • :fire-weapon --> "fw"
  • :switch-weapon --> "sw"
  • :set-crouch --> "sc"
  • :shove --> "sv"
  • :say --> "sy"
  • :pick-up --> "pu"
  • :put-down --> "pd"
  • :current-health --> "hc"
  • :max-health --> "hm"
  • :current-armor --> "ac"
  • :max-armor --> "am"
  • :current-location --> "lc"
  • :current-facing --> "fc"
  • :can-see --> "cs"
  • :radar --> "ra"
  • :what-is --> "wi"
  • :current-ammo --> "mc"
  • :range-finder --> "rf"
  • :check-inventory --> "ci"
  • :follow --> "fo"
  • :batch-range-finder --> "rb"
  • :pop --> "po"
  • :pause --> "pa"
  • :forget-all-tasks --> "fa"
  • :skip --> "sk"
  • :peek --> "pk"
  • :now --> "n"
  • :then --> "t"
  • :replace --> "r"

The client interacts with the server via four functions, each of which has a unique way of processing the return data. (If you are unsure which to use, try "send-and-get", as it is by far the most common.)

If you don't need the reply data 

client.core=> (doc send-and-forget)
-------------------------
client.protocol-one/send-and-forget
([quagent op scheduling args])
  Sends an op to the server and discards the results.
  Use the keywords in *codes* (see above) instead of 
  protocol zero codes. The args to the op should be in 
  a vector.
nil

If you want to get the data synchronously 

client.core=> (doc send-and-get)   
-------------------------
client.protocol-one/send-and-get
([quagent op scheduling args init f])
  Sends an op to the server and blocks until it returns.
  Replies are combined using the (init)ial value and 
  (f)unction supplied by the user.   Use the keywords in
  *codes* (see above) instead of protocol zero codes.
  The args to the op should be in a vector.
nil

If you want to get the data asynchronously 

client.core=> (doc send-and-get-later)
-------------------------
client.protocol-one/send-and-get-later
([quagent op scheduling args init f])
  Sends an op to the server and returns a future 
  that waits for the reply. Replies are combined
  using the (init)ial value and (f)unction supplied
  by the user.  Use the keywords in *codes* (see above)
  instead of protocol zero codes. The args to the op
  should be in a vector.
nil

If you want to dispatch on data individually 

client.core=> (doc send-and-watch)    
-------------------------
client.protocol-one/send-and-watch
([quagent op scheduling args init f wf])
  Sends an op to the server and processes replies with 
  an (init)ial value and (f)unction supplied by the user;
  when the (acc)ulator value changes, wf is called.
  wf must be a function that accepts 4 arguments,
  as in (fn [reference key old-val new-val] ...).  Use
  the keywords in *codes* (see above) instead of protocol
  zero codes. The args to the op should be in a vector.
nil

Notice that the room has a few items scattered around it; in order to find out where these are relative to the quagent, we need to define a function "scan-area" that takes two arguments (a quagent key and a radius) and returns a hash map of the positions of the items. Before trying to write the full function it is a good idea just to print out what the server is returning. 

client.core=> (send-and-get :Bob :radar :now [8000] nil (fn [prev data] (println data)))
(0 player 768.875366 90.065201 0.000000)
(2 player 32.000000 -90.000000 0.000000)
(72 info_player_deathmatch 32.000244 -90.000000 0.223811)
(74 quagent_item_treasure 572.168640 20.462269 0.901278)
(75 quagent_item_gold 375.085327 56.309933 1.374918)
(76 quagent_item_gold 1019.278626 42.455196 0.505915)
(77 quagent_item_treasure 697.711304 63.434952 0.739097)
(78 quagent_item_gold 905.141357 8.130102 0.569713)
(79 info_player_deathmatch 0.125000 0.000000 90.000000)
nil

The simplest implementation is to just use the basic radar op from protocol zero with a vector for an initial value and "conj" for the combination function. (Note the use of "pp" to pretty-print the previous result.) 

client.core=> (send-and-get :Bob :radar :now [8000] [] conj)
[("0" "player" "768.875366" "90.065201" "0.000000") ... (output-truncated)
client.core=> (pp)                                          
[("0" "player" "768.875366" "90.065201" "0.000000")
 ("2" "player" "32.000000" "-90.000000" "0.000000")
 ("72" "info_player_deathmatch" "32.000244" "-90.000000" "0.223811")
 ("74" "quagent_item_treasure" "572.168640" "20.462269" "0.901278")
 ("75" "quagent_item_gold" "375.085327" "56.309933" "1.374918")
 ("76" "quagent_item_gold" "1019.278626" "42.455196" "0.505915")
 ("77" "quagent_item_treasure" "697.711304" "63.434952" "0.739097")
 ("78" "quagent_item_gold" "905.141357" "8.130102" "0.569713")
 ("79" "info_player_deathmatch" "0.125000" "0.000000" "90.000000")]
nil

Partitioning these into a map is going to be a little more difficult as multiple positions will need to be stored at each key. However, we know already that the initial data structure needs to be a hash-map and the keys need to be the item type. 

client.core=> (send-and-get :Bob :radar :now [8000] 
                            {}
                            (fn [prev [_ item-type & pos]] ; data is destructured
                              (assoc prev item-type pos)))
{"quagent_item_gold" ("905.141357" "8.130102" "0.569713")... (output truncated) 
client.core=> (pp)
{"quagent_item_gold" ("905.141357" "8.130102" "0.569713"),
 "quagent_item_treasure" ("697.711304" "63.434952" "0.739097"),
 "info_player_deathmatch" ("0.125000" "0.000000" "90.000000"),
 "player" ("32.000000" "-90.000000" "0.000000")}
nil

This is progress but the data from the new replies is overwriting the previous results. To get the right behaviour, the "merge-with" function must be used to combine the maps. 

client.core=> (send-and-get :Bob :radar :now [8000] 
                            {}
                            (fn [prev [_ item-type & pos]]
                              (merge-with concat prev {item-type (list pos)})))
{"quagent_item_gold" (("375.085327" "56.309933" "1.374918") ... (output truncated)
client.core=> (pp)
{"quagent_item_gold"
 (("375.085327" "56.309933" "1.374918")
  ("1019.278626" "42.455196" "0.505915")
  ("905.141357" "8.130102" "0.569713")),
 "quagent_item_treasure"
 (("572.168640" "20.462269" "0.901278")
  ("697.711304" "63.434952" "0.739097")),
 "info_player_deathmatch"
 (("32.000244" "-90.000000" "0.223811")
  ("0.125000" "0.000000" "90.000000")),
 "player"
 (("768.875366" "90.065201" "0.000000")
  ("32.000000" "-90.000000" "0.000000"))}
nil

These positions can't be used as strings, however, and will need to be converted to doubles. 

client.core=> (send-and-get :Bob :radar :now [8000] 
                            {}
                            (fn [prev [_ item-type & pos]]
                              (merge-with concat 
                                          prev
                                          {item-type (list (map #(Double/parseDouble %) 
                                                                pos))})))
{"quagent_item_gold" ((375.085327 56.309933 1.374918) ... (output-truncated)
client.core=> (pp)
{"quagent_item_gold"
 ((375.085327 56.309933 1.374918)
  (1019.278626 42.455196 0.505915)
  (905.141357 8.130102 0.569713)),
 "quagent_item_treasure"
 ((572.16864 20.462269 0.901278) (697.711304 63.434952 0.739097)),
 "info_player_deathmatch"
 ((32.000244 -90.0 0.223811) (0.125 0.0 90.0)),
 "player" ((768.875366 90.065201 0.0) (32.0 -90.0 0.0))}
nil

The process of converting a sequence into doubles is so common that it has been included in the client as "seq->doubles". 

client.core=> (defn scan-area [quagent radius]
                (send-and-get quagent :radar :now [radius] 
                              {} 
                              (fn [prev [_ item-type & pos]]
                                (merge-with concat 
                                            prev
                                            {item-type (list (seq->doubles pos))}))))
#'client.core/scan-area
client.core=> (scan-area :Bob 8000)
{"quagent_item_gold" ([375.085327 56.309933 1.374918] ... (output truncated)
client.core=> (pp)
{"quagent_item_gold"
 ([375.085327 56.309933 1.374918]
  [1019.278626 42.455196 0.505915]
  [905.141357 8.130102 0.569713]),
 "quagent_item_treasure"
 ([572.16864 20.462269 0.901278] [697.711304 63.434952 0.739097]),
 "info_player_deathmatch"
 ([32.000244 -90.0 0.223811] [0.125 0.0 90.0]),
 "player" ([768.875366 90.065201 0.0] [32.0 -90.0 0.0])}
nil

You can now use the predefined "move" command to make the quagent walk to the second gold item (note that these positions are relative, not absolute). 

client.core=> (apply (partial move :Bob)
                     (take 2 (second ((scan-area :Bob 8000) "quagent_item_gold")))) 
[]

Now suppose that scanning the area takes an inordinate amount of time and blocking until it completes is no longer practical. By using the "send-and-get-later" function, the results can be computed in a new thread and dereferenced later. 

client.core=> (defn scan-area2 [quagent radius]
                (send-and-get-later quagent :radar :now [radius]
                                    {}
                                    (fn [prev [_ item-type & pos]]
                                      (merge-with concat
                                                  prev
                                                  {item-type (list (seq->doubles pos))}))))
#'client.core/scan-area2
client.core=> (def items (scan-area2 :Bob 8000))
#'client.core/items
client.core=> (type items)
clojure.core$future_call$reify__5508
client.core=> (future-done? items)
true
client.core=> @items
{"quagent_item_gold" ([375.085327 56.309933 1.374918] ... (output truncated)
client.core=> (pp)
{"quagent_item_gold"
 ([375.085327 56.309933 1.374918]
  [1019.278626 42.455196 0.505915]
  [905.141357 8.130102 0.569713]),
 "quagent_item_treasure"
 ([572.16864 20.462269 0.901278] [697.711304 63.434952 0.739097]),
 "info_player_deathmatch"
 ([32.000244 -90.0 0.223811] [0.125 0.0 90.0]),
 "player" ([611.700012 90.081947 0.0] [32.0 -90.0 0.0])}
nil

(Note that if you try to dereference a future before it completes, it will block the current thread until it does.)

Now let's suppose that every time the quagent reports finding an item, it should print out the distance that item. This can be accomplished with the "send-and-watch" function. 

client.core=> (defn scan-area3 [quagent radius]
                (send-and-watch quagent :radar :now [radius] 
                                nil
                                (fn [prev data]
                                 (rest data))
                                (fn [k r o n] ; short for "key", "reference", "old", and "new"
                                 (println "Item:" (first n) "Distance:" (second n)))))
#'client.core/scan-area3
client.core=> (scan-area3 :Bob 8000)
:watcher311
Item: player Distance: 611.700012
Item: player Distance: 32.000000
Item: info_player_deathmatch Distance: 32.000244
Item: quagent_item_treasure Distance: 572.168640
Item: quagent_item_gold Distance: 375.085327
Item: quagent_item_gold Distance: 1019.278626
Item: quagent_item_treasure Distance: 697.711304
Item: quagent_item_gold Distance: 905.141357
Item: info_player_deathmatch Distance: 0.125000

This function returns a watcher key that can be used to remove the watcher if desired. Note that this example only makes used the (n)ew argument.

Controlling Multiple Quagents

The simplest way to operate multiple quagents on the REPL is to wrap all commands with "future". (This is also the only function that will return directly.) 

client.core=> (future (move :bob 500 0))
#<core$future_call$reify__5508@12d7d02: :pending>
client.core=> (future (move :joe 500 0))
#<core$future_call$reify__5508@45ce17: :pending>

If you want to apply the same command to many quagents, some typing can be saved with the "pmap" function, which is exactly like "map" except the elements are processed in parallel. 

client.core=> (pmap #(move % 1000 0) (get-quagents))  
([] [])

Similarly, "pvalues" builds a lazy sequence of values. 

client.core=> (pvalues (scan-area :bob 8000) (move :joe 1500 180))            
({"quagent_item_gold" ([839.970581 160.375198 0.613917] ... (output truncated)
client.core=> (pp)
({"quagent_item_gold"
  ([839.970581 160.375198 0.613917]
   [703.026123 110.582535 0.733509]
   [142.61908 136.43396 3.618063]),
  "quagent_item_treasure"
  ([493.463074 159.83313 1.045044] [908.39563 139.15329 0.567672]),
  "info_player_deathmatch"
  ([1001.049011 -176.454605 0.007154]
   [999.580505 -178.285645 0.007165]),
  "player" ([1005.275879 -174.131836 0.0] [267.436554 -6.46857 0.0])}
 [])
nil

Scheduling Functions

Logic Programming

Sample Domains

Cave explorer

The movement planner included in the Clojure client is a basic maze explorer that uses value iteration to compute the next move. Cells that have been previously explored are given a reward of -0.04 and unexplored cells have a reward of 1.0.

Although this example was meant to showcase the ways in which Clojure's support for concurrency and parallelism could represent collaborative planning, unfortunately the incorrect type was used to store the shared data, and as a result the program is not thread-safe (though still completes with high probability). The problem arises in the value iteration algorithm, where the the function "max" is applied to a list that occasionally contains a nil value, which causes an exception (note that this exception doesn't show if the quagent is being run inside of a future and not a regular thread). This probably occurs because updates to the shared data structure are not coordinated, and as a result, if one quagent is looking up value that another has yet to write, then a nil value will be returned. This problem could be solved by using refs instead of atoms, since they can be coordinated without the possibility of race conditions.

Rovers

Automated planning, STRIPS domain, ICAPS '02.

For the Time domains, in the pddl file, replace the :duration value of navigate with 

:duration (= ?duration (/ (distance ?y ?z) 0.28))

and add the function 

(distance ?x - waypoint ?y - waypoint)

Now add distance data to the pfile. Example: 

 (= (distance waypoint0 waypoint0) 0)
 (= (distance waypoint0 waypoint11) 399)
 (= (distance waypoint0 waypoint10) 410)
 (= (distance waypoint0 waypoint9) 525)
 (= (distance waypoint0 waypoint7) 340)
 (= (distance waypoint0 waypoint8) 256)
 (= (distance waypoint0 waypoint5) 798)
 (= (distance waypoint0 waypoint6) 708)
 (= (distance waypoint0 waypoint2) 925)
 (= (distance waypoint0 waypoint1) 434)
 (= (distance waypoint0 waypoint4) 924)
 (= (distance waypoint0 waypoint3) 697)
 (= (distance waypoint11 waypoint0) 399)
 (= (distance waypoint11 waypoint11) 0)
 (= (distance waypoint11 waypoint10) 604)
 (= (distance waypoint11 waypoint9) 522)
 (= (distance waypoint11 waypoint7) 289)
 (= (distance waypoint11 waypoint8) 592)
 (= (distance waypoint11 waypoint5) 735)
 (= (distance waypoint11 waypoint6) 988)
 (= (distance waypoint11 waypoint2) 979)
 (= (distance waypoint11 waypoint1) 798)
 (= (distance waypoint11 waypoint4) 769)
.
.
.

Current Issues

  • Exploring the maze with multiple bots will occasionally cause one to hang. This is an issue coordinating multiple data structures in the maze explorer code, not in protocol one.