- What is the purpose of ATELstream?
Its purpose is to provide a fast, authenticated socket-level public distribution network for astronomical research applications. It builds on machine-readable, published, standardized information exchange formats, to permit machine-level (that is, not human level) interactions. It will accept XML objects, authenticating the client submitting them, and redistribute the XML objects to connected servers world-wide.
Its best use will be for users who require authentication (i.e. “I am who I say I am”) to a public research audience, who may not be identifiable in advance to the author, who may use of the information in a way not anticipated by the author, or make use of the information in ways that the author cannot easily do, to further a research goal. Many of these uses are described by the VOEvent effort. But some such messages include (in human language):
- “The X Telescope is right now being re-pointed to RA=Y, dec=Z, for J seconds”.
- “An optical transient at RA=Y, dec=Z was observed 10 seconds ago, with magnitude V=15. 30 seconds prior to that, at that position, the upper limit was V>23.”
- “Neutrino detector X is operating with sky coverage represented here. An anomalously high neutrino burst was detected at YY time UT.”
- Is ATELstream a perl script? No. ATELstream is a protocol and a service. The perl script is an implementation of the protocol which makes use of the service. ATELstream may be adapted into any language which can communicate via a socket layer using TCP.
- Is ATELstream a peer-to-peer service? No. The key part of ATELstream is that it makes use of authentication for authors submitting XML objects to the stream; this absolutely requires that all XML objects distributed by ATELstream begin at our servers. These XML objects are then re-distributed to connected remote servers using the “phone tree” construction of ATELsendto objects.
- What type of XML objects will ATELstream distribute?Content. Any. We wish to stress that there will be no restriction on the type, content, form or format of the XML object distributed by ATELstream now or in the future, with the exception that the XML object must be unique (i.e. the same XML object cannot be submitted twice). XML objects can be a described data point, or list of data points. They can be a short plain-language notice. They can be a test-message. We will not be filtering or enforcing any content standard on the XML objects distributed using authentication. However, we expect that all use of ATELstream is professional, and that the frequency of use not strain the service. Length. During the beta test period, that length is restricted to 10K ASCII characters. This will be reviewed as we pass out of beta testing. We expect, however, that lengths will not be longer than this, and if more space is needed for included data, that it is instead located at an included URI.Format. During the beta-test period, we have only one requirement: the XML object must be well-formed. Specifically, we will not be confirming that the XML object is validated, which means it conforms to a publicly described format, such as a DTD. Authors will appreciate that XML object validation is highly desirable for all users, and are strongly encouraged to submit pre-validated XML objects. We do not anticipate changing this policy as we move out of the beta-test period. Authors looking for a useful XML object schema to describe their astrophysical events should strongly consider VOEvent.
- If authors can submit gibberish XML, why would I believe anything which comes across ATELstream?
- Can I submit XML objects via ATELstream?
- How do I know XMLstream will be working next year?
- What is the “main node” of ATELstream?
- Please describe the process by which a remote server initially connects to the main node.
- Please describe the process by which an XML object is redistributed by ATELstream.
This is somewhat involved, so we break this down into steps:
- A client connects to the main node, presenting a credential for The Astronomer’s Telegram. This tells the main node that an XML object for redistribution is being submitted.
- An XML object is submitted by the client to the main node. The main node creates a hash of the XML object (using the perl md5_hex algorithm), and hands the hash back to the client for confirmation.
- The client runs the identical md5_hex algorithm on the XML object, confirms that the two hashes are identical, and responds to the main node “OK”, which lets the main node know that the XML object has been received uncorrupted. At this point, the main node disconnects from the client, commits the object to the database, prepares to redistribute the XML object through ATELstream. If the client gives any other response than “OK”, the main node discards the submitted XML object, responds with a statement to the client stating so, and disconnects from the client.
- The main node repackages the XML object within a new object (an ATELobject), which contains the credential information (author name, institution, email address, the IP address and hostname of the client computer which submitted the XML object); the hash of the XML object; a nested “ATELsendto” object (see below); and the XML object itself. This object is then redistributed to all connected servers by a peer-to-peer “phone tree” method using information in the nested ATELsendto object. The purpose of the peer-to-peer “phone tree” method is rapid distribution of the received XMLobject.
- The main node connects to remote servers, presents the token for that server (which lets the server know this is a legitimate ATELstream communication), and provides the ATELobject. The main node then disconnects.
- The remote server strips the hash provided by the main server and the submitted XML object, it performs the hash on the XML object and confirms that it matches the included hash. If they do not match, the remote server stops here (but remains ready for new connections).
- Following a successful hash match, the remote server parses the ATELobject and pulls out the ATELsendto object. The ATELsendto object contains three attributes: an IP address, a port, and a valid token, all corresponding to a remote server which had checked in to the main node. The top-level child ATELsendto is the remote sever itself. Also, each ATELsendto can have nested ATELsendto objects. The remote server strips off the ATELsendto object from the ATELobject, removing its own ATELsendto object, and parses the children to its own ATELsendto object. Finally, it repackages the ATELobject for each of the children remote servers, and performs a connection to each child in turn, first presenting each child remote server with its valid token, and then providing the full ATELobject.
- After redistributing the ATELobject to all next-generation children, the remote server then connects to the main node, and confirms receipt of the ATELobject, by presenting its token, and the hash of the received ATELobject to the main node, before disconnecting. At this point, the remote server returns to “listening” mode, and handles the provided XML object (e.g. feeding it to a telescope queue, awaking someone via text message, tweeting it, adding the XML object to a library, and so on — whatever function the remote server wishes, based on the XML object properties).
- Following a short period after the initial distribution (estimated to be seconds) the main node examines the list of connected servers and confirms that all have sent receipts. For servers which had not sent receipts, the main node attempts a direct connection and sends the ATELobject directly.
- What if servers are bad citizens, and don’t send receipts to the main node, or continually fail to redistribute ATELobjects to dependent children?
- I’d like to be able to get submitted objects from three weeks ago. Can I?
During beta testing, whenever you start up a server, the server will respond with connections providing objects submitted within the past 24 hours. This will likely carry over, once the beta test period is complete. There are no plans to make the entire set of ATELstream objects indefinitely queryable from the main node. ATELstream is primarily an authentication and distribution point, and not an archive.
- I notice you published your client and servers as perl scripts. I prefer using python/C/C++/Java/COBOL. Can I simply try to duplicate the functionality and re-program the processes I see in those perl scripts?
Absolutely. And we ask that you notify us (perhaps through a comment below) with a link to the code, so that others can make use of it too.
- What is this “beta-test” period you keep referring to?
ATELstream launched in “best-test” mode in March 2011. The purpose of beta-test mode is to bring the initial design and execution of the main node, clients, and server scripts into contact with real-world applications (and problems), largely as a “toy” to play with. We expect during this period that there may be periods of server outage; that the rate of “gibberish” posts will be high and that we and the community will learn a lot about how ATELstream should operate in deployment. During this period, we recommend against using ATELstream for any “mission-critical” applications, but we strongly recommend testing its use within your real-world applications.
- Can I send test messages through ATELstream? Yes. See the question above regarding “content.” We wish to stress that we will make absolutely no constraint on any type of content distributed through ATEL. This permits test messages.
- Should I stop submitting posts to The Astronomer’s Telegram and instead submit posts to ATELstream?
No. The Astronomer’s Telegram is best for distributing human readable information, while ATELstream is best for distributing machine readable information. The Astronomer’s Telegram serves the need of distributing information which must come to the attention of people. ATELstream serves the need of distributing information which must quickly be responded to, by computers.
- Why are you using TCP instead of HTTP? I can’t get TCP behind my firewall. HTTP is slow, TCP is much faster. The design specification for ATELstream is for speed of delivery. If it is important for you to receive them, then you should either open up a port through your firewall, or you should write a relay server on a machine which is in front of your firewall.
Authors — who submit XML objects for redistribution using their approved credential — are 100% responsible for their submitted XML object. The Astronomer’s Telegram is performing the authentication function: we vouch that the author is who they say they are, and therefore you can trust the XML object with precisely the same amount of trust as if you have for the author.
To distribute an XML object through ATELstream, you must submit it to the main node using an approved credential from The Astronomer’s Telegram. During the beta test period, however, you may submit XML objects with a username “firstname.lastname@example.org” and password “test”. These will be distributed without an author name, email, or institution, but with the originating IP, port and hostname.
The Astronomer’s Telegram has been operating continuously since December 1997. We are committed to the same level of longevity and reliability for ATELstream. While the long-term provision of ATELstream will depend on its level of usage, we believe we can easily commit to keeping ATELstream active for at least 10 years — through January 2021 — so that those who are building automated systems can consider ATELstream’s utility in their design plans. We plan to develop this capability to conform with usage.
ATELstream’s main node — the server which manages everything — is at astronomerstelegram.org:33331. This node manages all requests of networked clients and servers. When a new server starts up, it checks in there. When a client submits an XML object, it is submitted there. When an XML object is redistributed, the redistribution begins from there.
A remote server, on start-up, “pings” the main node, and obtains a “token”. This token, thereafter, will be presented (either by the main node, or another computer in the network) to the remote server, which lets the remote server know that the communication which follows originated from the main node. At appropriate times, another server at the IP address of the main node will update this token.
There is a process of network optimization which is opaque to users. ATELstream uses token updates to determine network lagtimes, to probe for servers which have disconnected from the network; and it also uses past re-distribution behavior (keeping track of which servers submit to which other servers) to shape the network.