1 Take a look in INSTALL and install all the dependencies.
3 Look at config.py and make sure it's pointing at your mongodb
5 Kick off the HTTP API server by running: ./magic_httpd.py
7 That's it! You're up and running! Okay, so you probably want to do stuff
8 with it at all. There is a simple shell client to talk to the HTTP API for
9 testing. For production, you'll want to use mudpuppy (or your own agent)
10 to automate the individual items.
13 OKAY, IT'S RUNNING, SO NOW SHOW ME MORE THAN JUST A BLINKING CURSOR!
15 By default, make-magic is using an example set of possible items to do
16 read from doc/sample_items.json, which is a simple list of steps for getting
17 out of bed in the morning and going to work. When you make a new task from
18 this, the only requirement that it recognises is 'coffee'. If you say you
19 require coffee, the steps to make it and drink it will be included.
21 Let's get started. First, we can list the tasks that are currently being
22 handled by make-magic:
24 davidb@kelp:~/src/make-magic$ ./mclient.py tasks
26 davidb@kelp:~/src/make-magic$
28 Not very interesting. Let's create a task, and say we require coffee.
30 davidb@kelp:~/src/make-magic$ ./mclient.py task_create '{"requirements": [ "coffee" ]}'
34 "state": "INCOMPLETE",
35 "name": "TaskComplete",
41 (... many more items... )
44 "state": "INCOMPLETE",
45 "name": "make_coffee",
55 "uuid": "1ede91f0-6b39-4da9-8fe6-cc0b028ed349",
59 davidb@kelp:~/src/make-magic$
61 mclient is pretty simple. Most of the time it will just talk the JSON that is
62 part of the (vaguely RESTful) HTTP API. make-magic has created a new task
63 based on the requirements (in this case make_coffee and drink_coffee are in
64 there because we said we required coffee). It's also allocated a uuid for the
65 task. If we had wanted to, we could have added more key/value pairs, and they
66 would have been added into the task metadata.
68 If we hadn't told it we required coffee, it would have removed any reference to
69 it, and nothing would depend on the removed items:
71 davidb@kelp:~/src/make-magic$ ./mclient.py task_create '{"requirements": []}'
75 (... items in here but no making or drinking coffee ...)
80 "uuid": "301b925c-cf35-4195-8bfa-0fa41ccaf8c8",
84 davidb@kelp:~/src/make-magic$
87 Let's list the tasks again:
89 davidb@kelp:~/src/make-magic$ ./mclient.py tasks
91 "1ede91f0-6b39-4da9-8fe6-cc0b028ed349",
92 "301b925c-cf35-4195-8bfa-0fa41ccaf8c8"
94 davidb@kelp:~/src/make-magic$
96 Now it's showing the UUID for the tasks we just created. If we want, we can now
97 ask for all the information associated with a task with:
99 davidb@kelp:~/src/make-magic$ ./mclient.py task 1ede91f0-6b39-4da9-8fe6-cc0b028ed349
101 Okay, so there are a whole lot of things that need to be done, but some things
102 depend on other things. What we really want is a list of things that need to be
103 done, but which haven't had their dependencies satisfied yet. We can do this with:
105 davidb@kelp:~/src/make-magic$ ./mclient.py items_ready 1ede91f0-6b39-4da9-8fe6-cc0b028ed349
108 "state": "INCOMPLETE",
112 davidb@kelp:~/src/make-magic$
114 At the moment, the only thing that we can do at the moment is to wake up. If you
115 were inclined to look through all the items and follow their dependencies, this is
116 because you're not going to be able to get out of bed till you're awake, and you're
117 not going to be able to do the rest of getting ready for work unless you're in bed
120 What we'll do is change the state from INCOMPLETE, to IN_PROGRESS, and finally to
123 davidb@kelp:~/src/make-magic$ ./mclient.py update_item_state 1ede91f0-6b39-4da9-8fe6-cc0b028ed349 wake_up INCOMPLETE IN_PROGRESS
125 davidb@kelp:~/src/make-magic$ ./mclient.py update_item_state 1ede91f0-6b39-4da9-8fe6-cc0b028ed349 wake_up IN_PROGRESS COMPLETE
127 Before a client does any work on an item, it first sets it's state to IN_PROGRESS so that
128 other workers know not to also try and do it. When the client has finished successfully it
129 sents the state to COMPLETE.
131 (Some notes on concurrency: update_item_state is pretty much the only time the client is doing
132 anything other than a single HTTP request passing what it gets from the command line.
133 The reason is that you will likely have multiple agents (or a single agents with
134 multiple threads or processes) looking for items to do, and then moving them to
135 IN_PROGRESS to do them. To guard against race conditions, in this case the state
136 will only be changed if it already matches the one that we've told it we're changing
137 from (the server enforces this atomically), and the client also passes a random token to the
138 server that will only come back if it's request was the one that succeeded)
141 Now that we've gotten out of bed, if we check to see what items are ready again:
143 davidb@kelp:~/src/make-magic$ ./mclient.py items_ready 1ede91f0-6b39-4da9-8fe6-cc0b028ed349
145 it will show the next step is 'get_up'. Let's complete that as well:
147 davidb@kelp:~/src/make-magic$ ./mclient.py update_item_state 1ede91f0-6b39-4da9-8fe6-cc0b028ed349 get_up INCOMPLETE IN_PROGRESS
148 davidb@kelp:~/src/make-magic$ ./mclient.py update_item_state 1ede91f0-6b39-4da9-8fe6-cc0b028ed349 get_up IN_PROGRESS COMPLETE
150 (Yes, this gets boring pretty quick, but the whole point is to automate all this
151 stuff. Mudpuppy, which is also posted on anchor's github account, will do all this
152 dealing with state stuff for you and let you write simple modules to do the steps)
154 Now let's check what items are ready to go, because it's slightly more interesting
156 davidb@kelp:~/src/make-magic$ ./mclient.py items_ready 1ede91f0-6b39-4da9-8fe6-cc0b028ed349
159 "state": "INCOMPLETE",
160 "name": "make_breakfast",
166 "state": "INCOMPLETE",
167 "name": "make_coffee",
173 davidb@kelp:~/src/make-magic$
175 Now that we're out of bed, there are two things available to do. Make breakfast, or
176 (because we said we needed it), make coffee. The important thing to note is that
177 both of these can be done at the same time! Both of them have had all their dependencies
178 completed; if one depended on the other from finishing, it wouldn't show up in the
179 list. One of the cool things about make-magic is that you can do multiple steps at the
180 same time, and make-magic will keep track of which items are completed, which ones
181 are needed by other items still, and figure out on the fly what has to be done next.
183 You can now (if you desire) go through and do all the tasks in order. Myself, I'd recommend
184 getting something like mudpuppy to automated them, which is indeed the whole point.
185 See: https://github.com/anchor/mudpuppy
187 There is a single special item that is created for automatically for each task, and
188 that's called TaskComplete.
190 davidb@kelp:~/src/make-magic$ ./mclient.py item 1ede91f0-6b39-4da9-8fe6-cc0b028ed349 TaskComplete
192 "state": "INCOMPLETE",
193 "name": "TaskComplete",
199 TaskComplete depends (indirectly) on every item in the task. If you ask make-magic
200 for items that are ready, and it sees that the only item ready to go is TaskComplete,
201 the server will set it to COMPLETED itself, and return back an empty list of things
207 There is actually a few more useful things you can do. You can add and update items
208 in the task metadata:
210 davidb@kelp:~/src/make-magic$ ./mclient.py metadata 1ede91f0-6b39-4da9-8fe6-cc0b028ed349
215 "uuid": "1ede91f0-6b39-4da9-8fe6-cc0b028ed349",
218 davidb@kelp:~/src/make-magic$ ./mclient.py update_metadata 1ede91f0-6b39-4da9-8fe6-cc0b028ed349 '{"mornings_are": "meh"}'
219 (... previous metadata contents...)
222 "mornings_are": "meh",
226 "uuid": "1ede91f0-6b39-4da9-8fe6-cc0b028ed349",
229 davidb@kelp:~/src/make-magic$
232 We use it for things like saving the IP addresses that we've allocated to a
233 server so that other items later on have easy access to it (like the ones
234 setting up networking) without having to store it somewhere else.
236 You can also add metadata to items in a very similar way:
238 davidb@kelp:~/src/make-magic$ ./mclient.py item 1ede91f0-6b39-4da9-8fe6-cc0b028ed349 get_up
240 "_change_state_token": 18106579636852,
247 davidb@kelp:~/src/make-magic$ ./mclient.py update_item 1ede91f0-6b39-4da9-8fe6-cc0b028ed349 get_up '{"bed was": "comfy", "sleep": "good"}'
249 "_change_state_token": 18106579636852,
265 "_change_state_token": 18106579636852
267 davidb@kelp:~/src/make-magic$
269 You can use this for pretty much anything. If automation fails and you have to
270 change it to the FAILED state, you add in debugging information as to why for
273 Now you've gone through this, it's probably going to be more interesting to
274 define your own items (we have hundreds in our own production environment).
276 Rather than just having stuff filtering on a single requirement, you can filter
277 an individual item on many different ones, e.g.:
280 "name": "do_something",
281 "if": "(os.debian | os.rhel5) & hardware.vm & ( ! support_level.basic )",
283 "reboot_debian", "reboot_rhel"
287 would only turn up in a task if the requirements included "os.debian" or "os.rhel5",
288 also included "hardware.vm", but didn't include "support_level.basic".
290 It might at first seem a bit seem a bit weird that it's depending on both
291 reboot_debian and reboot_rhel, but the definition of reboot_debian
292 will almost certainly include "if": "os.debian" at the very least, and
293 similar for RHEL; Any dependencies that are filtered out by their own
294 'if' entries will also be removed from any dependencies when a task is
297 This works much better than you would first expect; It also gives you
298 the ability to build a complex list of item dependencies without having
299 to explicitly define every single permutation of tasks that can be
300 generated from requirements (in our case we would die of old age before
301 being able to define them by hand). This is part of what makes