Introduction
This page contains a description of DamageControl's XML-RPC interface.
It is our intention to agree on this API with other Continuous Integration systems such as CruiseControl, CruiseControl.NET and BuildBot.
If we (developers of these CI systems) agree on the API, clients that are written to conform to it will be usable across all CI systems. Please comment at the bottom of this page if you want to participate in the consolidation of the API. You must sign up for an account here if you wish to comment.
The reason why we (DamageControl team) are suggesting XML-RPC is:
- It is platform neutral, and clients can be written for many platforms.
- It is simpler than e.g. SOAP, but should be powerful enough for all CI systems.
(We are considering moving to SOAP though, see comments at the very bottom).
Clients
There are several clients out there (some stable, some under development or planning) that currently use a "proprietary" API (i.e. won't work with other CI systems than the ones they are designed for). We can fix this together and have much better interop! The clients we know of are:
Client/Protocol |
Description |
|---|---|
A system tray app for Windows that monitors builds on CC.NET. |
|
An API that allows remote management of CC (Java) |
|
A CruiseControl Java Swing client integrated with IntelliJ IDEA |
|
CruiseControl Yahoo Messenger |
A Y! bot that watches CruiseControl builds |
DamageControl IDEA plugin |
Similar to Dashboard, but uses DamageControl's current XML-RPC API |
DamageControl Web Admin |
A web application that allows remote management and monitoring of DamageControl |
DamageControl IRC |
An IRC bot that notifies an IRC channel about DamageControl's status |
DamageControl Jabber |
A Jabber bot that notifies an IRC channel about DamageControl's status |
Again, the goal of this XML-RPC effort is to make all these clients conform to the same API. More on how to achieve this in XML-RPC Bridges.
Public and Private APIs
Some operations should be open for all. This includes all operations that don't modify any state on the CI system, but merely provide querying functionality. These operations should be part of the public API.
Other operations should be restricted to administrators or project developers. These are operations that modify the CI system's state.
The suggested public and private APIs below have been gathered by looking at DamageControl, CruiseControl and CruiseControl.NET's various remote APIs.
Public API
Usually unauthenticated. Accessible through http://server:4712/private/xmlrpc
Syntax |
Description |
Required? |
|---|---|---|
|
Get all project names |
Yes |
|
Get project names corresponding to currently running builds or an emty array if server is idle |
Yes |
|
Get an array of struct. Each struct represents a build. The fields of the build struct is defined below |
Yes |
|
Last completed build, null if no such project |
Yes |
|
Additional info for a build. |
No |
Private API
Usually authenticated and encrypted by an Apache proxy. Accessible through http://server:4712/private/xmlrpc. Can be unauthenticated in closed (corporate) environments (although not recommended for big corps).
Syntax |
Description |
Required? |
|---|---|---|
|
Requests a build, scm_timestamp_utc has to be formatted according to #Timestamp format. |
Yes |
|
Pauses a currently ongoing build. |
No |
|
Resumes a paused build. |
No |
|
Gets an array of all projects managed by the CI server. |
No |
|
Registers a new (or updates an existing) project to be managed by the CI server. |
No |
|
Registers new (or updates existing) user information. |
No |
Data formats
Build struct
Represents a build result (in the past or currently running)
XML-RPC type - Field Name |
Description |
Required? |
|---|---|---|
|
The name of the project |
Yes |
|
Status of the buld |
Yes. One of "IDLE", "SUCCESSFUL", "FAILED", "QUEUED", "BUILDING", "PAUSED", "CHECKING OUT" |
|
array of Modification struct |
Yes |
|
The time the build was requested, in UTC according to the SCM machine |
Yes |
|
The time the build started, in UTC according to the CI machine |
Yes |
|
The time the build ended, in UTC according to the CI machine |
Yes |
|
The label |
No. Should only be set for SUCCESSFUL builds, but even then is optional |
|
URL pointing to a build-specific web page with more detailed info |
No. |
Modification struct
This struct represents a changed file.
XML-RPC type - Field Name |
Description |
Required? |
|---|---|---|
|
relative path to the modified file. '/' as path separator |
Yes |
|
SCM id of the developer |
Yes |
|
The commit message |
Yes |
|
The current revision of the file |
Yes |
|
The timestamp of the file, in UTC according to the SCM |
Yes |
|
What happened to the file since the last revision |
No. If present, should be one of the following: "MODIFIED", "ADDED", "MOVED", "REMOVED" |
Project struct
Represents a project to be managed by the CI system
XML-RPC type - Field Name |
Description |
Required? |
|---|---|---|
|
The name of the project |
Yes |
|
Location of the SCM (We might need a new struct to support various SCMs) |
Yes |
|
Command line to execute in order to build the project |
Yes |
|
Where to email build results |
Yes |
|
Whether to mail on successful builds |
Yes |
|
URL pointing to the main page for build results |
No |
|
URL pointing to an RSS feed for builds |
No |
User struct
Represents a user (a developer with commit access to the SCM).
XML-RPC type - Field Name |
Description |
Required? |
|---|---|---|
|
The SCM user id |
Yes |
|
Full name of the user |
Yes |
|
User's email address |
Yes |
|
User's phone number |
No |
|
User's Y! id |
No |
|
User's msn id |
No |
|
User's jabber id |
No |
Timestamp format
yyyyMMddhhMMss
22 Comments
Hide/Show CommentsApr 21, 2004
Aslak Hellesøy
Anyone interested in starting a new project (on SourceForge maybe) called ci-clients? This project could host all the clients and admin interfaces, as well as provide a wiki and mailing lists.
Apr 22, 2004
Chris Stevenson
In Private API we need a kill_build* to terminate the current build. This is important since quite often a test will stick in some horrible timeout situation if for example the db goes away, and we can want to kill it, fix the db and restart the builds.
Come to think of it - maybe we want a stop_all_builds and start_all_builds, or is that what pause is for?
Personally I think pausing a build is more trouble than its worth, and could be quite fragile. Killing the current build and restarting it later seems better.
*maybe even a kill_build_2 - OK so that's a pathetic pun.
Apr 22, 2004
Mike Roberts
CCNet also has a 'Web Dashboard' - example at http://ccnetlive.thoughtworks.com/dashboard/, which would be a client of the public API
Apr 22, 2004
Mike Roberts
re: Public API - Why do we need user info? Also, the public API for CCNet also gives more details of the current state of a project (i.e. what its currently up to 'right now') It would also be good for the public API to get a styled build report, and the original build log (that's what we'll be adding to CCNet once its 'normal' webapp becomes more client/server based)
Apr 22, 2004
Mike Roberts
re: Private API. I think project configuration related methods / structs could be tricky to standardize on. I think as a first cut it would be much better to agree on more atomic functions ('start', 'stop', 'force', etc.)
Apr 22, 2004
Jon Tirsen
<Mike Roberts>
re: Public API - Why do we need user info?
</Mike Roberts>
For example, clients that integrate with Yahoo or email or whatever can use that information for spamming people that have modifications in a build.
<Mike Roberts>
Also, the public API for CCNet also gives more details of the current state of a project (i.e. what its currently up to 'right now')
</Mike Roberts>
DamageControl handles this a little bit differently. Basically the status is always what the build is up to currently (checking out, building etc) and will switch to SUCCESSFUL or FAILED when the build is completed. That is why the API has two functions get_current_build (what's currently going on) and get_last_completed_build (what happened to the last build).
Personally I think there is only one status for a project and separating them into two different ones (activity and status) is confusing. But that's just me.
Apr 22, 2004
Mike Roberts
<JT>
For example, clients that integrate with Yahoo or email or whatever can use that information for spamming people that have modifications in a build.
</>
Yup - sorry, just realised that, and I agree. At first I thought it was a security administration thing.
<JT>
DamageControl handles this a little bit differently. Basically the status is always what the build is up to currently (checking out, building etc) and will switch to SUCCESSFUL or FAILED when the build is completed. That is why the API has two functions get_current_build (what's currently going on) and get_last_completed_build (what happened to the last build).
Personally I think there is only one status for a project and separating them into two different ones (activity and status) is confusing. But that's just me.
</JT>
OK, I think there's a difference between project details (Name, Web site), project activity (building, checking out, last build date, etc.) and build results('failed', 'passed', 'label', 'date', report, log). I think build results should always relate to a completed build, and project activity relates to the current incomplete build.
Maybe we could think about having something like:
Server
{
string[] GetProjects();
string GetServerLog();
ProjectDetails GetProjectDetails(string projectName);
ProjectActivty GetProjectActvity(string projectName);
BuildResults GetBuildResults(string projectName, date buildDate)
string GetBuildReport(string projectName, date buildDate)
string GetBuildLog(string projectName, date buildDate)
// Then have convenience methods for 'last good build', 'current running projects', etc, maybe?
}
BTW, we need to key build results by date since CCNet (at least) doesn't key failed builds by label.
So, a question I have - if we are writing a generic API, should we actually include the 'convenience' methods? They would allow less round-trips between client and server, but would give a larger API to implement.
I apologise for this being .NET and CCNet centric - I'm brainwashed!
PS - good idea with the confluence inline referencing...
Apr 22, 2004
Aslak Hellesøy
Re: Chris Kill Build
I like that. Pausing/resuming seems contrived.
Re: Mike It would also be good for the public API to get a styled build report
I think it should be up to the client(s) to generate a styled report from the raw data provided by the API. (As suggested in the XML-RPC bridges page, I think e.g. a web app could be implemented as an XML-RPC client rather than hooking into the core of the CI system).
Re: Mike Convenience methods
I think they will probably be needed, but I think we should focus on the low level methods for now. It'll be easier to spot what good high level APIs should look like later on.
Re: Mike About being brainwashed
That's good! Brings a different perspective
I think we can come up with a ton of desired methods here, and no single CI system will implement all. When we set up the SF project for this, we can keep a matrix of the API that says what CI system supports a particular method, and whether the method is mandatory, recommended or optional.
As a start, maybe we should set up a test bench? I mean a dumb XML-RPC server that honours the API and provides some test data (but actually does nothing or very little more). Would be used to test various clients and could be hosted at e.g. SF or Codehaus. Should be east to knock up in Ruby during a geek night or two
Apr 22, 2004
Mike Roberts
<AH>
I think it should be up to the client(s) to generate a styled report from the raw data provided by the API...
</>
OK, you're right in that GetBuildReport shouldn't be mandatory. I put it in since we are going to add this behaviour to the CCNet server at some point. Right now we always style it in the web app (so client side effectively) but this is too slow for projects with big builds.
<AH>
I think we can come up with a ton of desired methods here...
</>
I think it would be great if we could start with deciding a core set of mandatory methods. From my set, I'd be really happy with 'GetProjects', 'GetProjectDetails', 'GetProjectActivity', 'GetBuildResults', 'GetBuildLog'. In theory, the 2 project ones could be combined (but I'd rather not since I think they mean different things), and 'GetBuildLog' could be dropped.
Apr 22, 2004
Mike Mason
Source control systems like Subversion and Perforce support the notion of "change sets", where a single logical change, with a single commit message, is associated with a group of files. This is in contrast to the way CVS, VSS and StarTeam work, where each file has independent history and changes are not grouped together.
I'd like to see the modification struct contain an array of filenames that have been modified, rather than just a single file, so we can support changesets in future. For now, most implementations would return arrays of size 1 with a single file in them.
Apr 22, 2004
Aslak Hellesøy
I had a chat with Erik Doernenburg about this stuff and he said "Go for SOAP/WebServices rather than XML-RPC". His reasoning was that although it's a tad more complicated, it is something more developers are likely to know. In addition to that, WSDL gives a non ambiguous definition of the API and structs (AFAIK - I have little SOAP experience). So I am all for going for SOAP. Whatcha think?
Mike M: Supporting changesets as you describe it should be a no-brainer
Mike R: It's great that you have styled report functionality in CC.NET. However, wouldn't it be possible to factor this out of the CC.NET core and make it work in isolation by pulling data from XML-RPC/SOAP? That way we could all benefit from it. (Likewise, DC can generate graphs with gnuplot - see http://docs.codehaus.org/display/PICO/Statistics). We could also externalise this and make it depend on the public API only.
=== And here is the good part ===
1) We create a new SF project called Continuous Integration Integration (or something similar - bring on your lame name suggestions too)
2) We set up a wiki and a mailing list so that everybody feels at home and invite CC, CC.NET, BuildBot and DC developers.
3) We develop a reference implementation of the API and expose it via SOAP or XML-RPC (we should decide on one protocol only to make it easier to be compliant)
4) The reference implementation is a dummy, but it implements the entire API and provides dummy data without actually building anything.
5) It also exposes some "test" methods (such as checkin) just to simulate a real environment.
6) All clients that wish to conform to the API can be tested against this app.
7) This app is deployed somewhere it can be accessed publicly.
8) We also develop a server compatibility kit. This is a command line client that can be configured to access a server on any given URL and runs a set of tests and generates a compatibility report.
9) Now we have a complete test/compatibility environment for both servers and clients.
10) We do this in an agile way. We all have an idea of what we want based on this document, but incremental development of this suite will allow us to come up with a solid API.
I suggest we develop this compatibility kit in Java/GLUE for the following reasons:
a) .NET is hard to host on an OSS server like SF or Codehaus. The host must be a place where we can give all interested people access.
b) Dynamic languages like Ruby can't generate WSDL from code because there is not enough metadata in the code to do that. (Correct me if I'm wrong).
c) Java is easy to host, and with GLUE (which AFAIK allows free use for OSS) can generate WSDL from source code.
(P.S. If we stick to XML-RPC, WSDL isn't an issue and in that case I'd prefer to do it in Ruby).
Those of you who are interested in this should raise your hand. If someone wants to take on the responsibility as a project admin it would be even better. (Means registering the project and setting up infrastructure).
Apr 22, 2004
Mike Roberts
re: SOAP vs. XML-RPC. I hadn't tried XML-RPC before, but its really easy (using XML-RPC.NET library). I got a client and server working this evening. I'd say go with XML-RPC
re: reporting - yeah, that maybe an idea. Problem is that its all dependent on the format of the log file, which is probably going to change for each CI implementation and build tool that they use.
re: project name. no idea.
OK, I've tracer bulletted an implementation in .NET, both client and server, running against CCNet. Its running on CCNetLive, so here are some URLs
1) server, listing projects running on CCNetLive - http://ccnetlive.thoughtworks.com/XMLRPCWebService/XMLRPC.rem
2) client - simple web page hitting (1) - http://ccnetlive.thoughtworks.com/XMLRPCWebService/BasicTest.aspx
3) Source code is in CCNet CVS tree, or in the latest source zip at http://ccnetlive.thoughtworks.com/CCNet-builds/0_6_0_210/CruiseControl.NET.source.zip
I'm happy for us to use CCNet as a ref impl, but if you want to host in Java and Ruby somewhere go ahead and I'll try to keep the CCNet Impl up to date.
Apr 23, 2004
Marco Jansen
Great ideas guys!
I would like to stretch the idea by also including a public publishing interface.
There will be at least two types of outgoing interfaces. One stream are messages in case a build finishes (either successful or a failure). This message contains the structs as described by Aslak and also the build artifacts. You can compare this with the Publishing facility in CC.
The other stream is a continuous stream with status information. Basically a tail of the logfile. With this stream you can do something similar that you can do with LumberMill and Log4J: you can build a nice GUI to track the status of the build server. Especially when the build process is long it gives you more insight information about what is happening.
When continuous integration systems share this publishing interface, you can define a common publishing tool to send e-mails, IM's, text-messages, FTP artifacts, scp artifacts etc. Also it is easy for people to define their own listeners.
The nice thing about a publishing interface is that you can reduce the load of the CI-server. So instead of all the developers pinging every 15 seconds for the status via the public API, you just send one message.
Marco.
Apr 23, 2004
Aslak Hellesøy
I will file a project request at SourceForge today. The unix name will be "cii", the real name "Continous Integration Integration". Holler if you have a better name.
As soon as the project is set up with a mailing list, I'll post an update here. Then we can start to work on the reference implementation and documentation.
It is important that the reference implementation has the following characteristics:
1) It should always be live so people can test against it.
2) It should be continuously integrated. A commit should deploy it anew.
3) It should be really simple to change and understand.
4) Self documenting (ideally).
So I guess it's going to be a Ruby+XML-RPC thing for now.
Re: Marco / publishing API
Great idea! You now have an area of responsability in CII
Apr 23, 2004
Mike Roberts
re: name - maybe 'OpenCI' ?
re: project - I'd like to add .NET interfaces (to be implemented by servers) and client proxies, so can we think about source tree directory structure?
e.g. http://cvs.sourceforge.net/viewcvs.py/ccnet/ccnet/project/XMLRPCWebService/XMLRPC/Server.cs?rev=1.1&view=auto http://cvs.sourceforge.net/viewcvs.py/ccnet/ccnet/project/XMLRPCWebService/Client/ServerProxy.cs?rev=1.1&view=auto
Apr 23, 2004
Aslak Hellesøy
OpenCI is already a project. I thought of CIA (Continuous Integration API), but that was taken on SF. So I went for FBI instead. When I had registered the project at SF I came to think "What does FBI mean?". I came up with this:
So folks, put up your SF user ids so I can add you as committers and lets get this Friggin Ball Implemented. (Sorry, it's Friday and I already had a pint).
Apr 23, 2004
Mike Roberts
Funky Build Integration (although Royle says it should be 'fulgent', whatever that means...)
my SF ID's 'mikeroberts'
Apr 26, 2004
Jon Tirsen
I agree, we should probably move to SOAP for the reasons explained by Erik Doernenburg. XMLRPC is simple, but I've tried a couple of .NET implementations now, and all of them are having some kind of interop-problems with the Ruby implementations. I've gotten it working using various work-arounds, but I think things could be more stable with SOAP.
Apr 26, 2004
Mike Roberts
Do you really think compatibility is going to be easier with SOAP? I know in the 'old' days that trying to get Apache SOAP and .NET SOAP working together was painful, but maybe things have got better. I'll try and get a SOAP service up on CCNetLive equivalent to the XML-RPC one I did last week - maybe you could try writing a Ruby client against it once its available?
Apr 27, 2004
Aslak Hellesøy
I guess the choice on XML-RPC/SOAP relies on our findings regarding different platform interop (.net, java, ruby, python, etc). Under any circumstance I think we should pick only one of them for this standard. Otherwise it would be too much pain to implement clients and servers (they'd have to be compatible with both transport protocols).
The SF project was rejected because I failed to provide enough info. I'll have another go....
Apr 27, 2004
Aslak Hellesøy
Watch this space: http://sourceforge.net/tracker/index.php?func=detail&aid=943416&group_id=1&atid=200001
Apr 28, 2004
Aslak Hellesøy
o The FBI project has been created: http://sourceforge.net/projects/fbi/
o The previous description: http://docs.codehaus.org/display/DC/Remote+API?showComments=true#comments (contents will be moved to FBI's docs)
o Some initial code is checked in to the 'fbi' CVS module
o Mailing list created (fbi-devel@lists.sf.net) - Let's continue discussion there now.
o All CC/CC.NET/DC/BB developers are invited to join (just let us know your SF id on fbi-devel@lists.sf.net)